# HTTP-Hygiene
> How to make your server tell an agent the truth: soft 404s, redirect chains, refused HEAD, missing charset, and 429 without Retry-After.
Diese Prüfung trägt 4 % des Scores.
*Diese Leitfäden zur Behebung sind in englischer Sprache verfasst.*
A person reads your error page and understands it. An agent reads the status line and believes it. Everything in this check is a place where your server tells the truth to one of them and not the other.
---
## A page that does not exist answering 200
The check requested a URL that cannot exist and got `200 OK`. To an agent, every URL on your site now exists, including the ones it invents. It cannot tell a real product page from a typo, so a wrong guess is indistinguishable from a right one and it may quote a page that was never there.
```
# what it should say
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=utf-8
# what a soft 404 says
HTTP/1.1 200 OK
```
In a single-page app this is usually the catch-all route answering before the router decides. Render the 404 view with a 404 status.
---
## A missing page redirecting to the homepage
Worse than a soft 404, because it also destroys the URL. An agent following a broken link lands on your homepage with a `200` and no signal that it did not arrive where it was going, and may cite the homepage as the source of something it never said.
---
## A missing page answering 403
`403` means "this exists and you may not". A polite agent will back off from the whole area rather than conclude the URL was wrong. Answer `404` or `410` for things that are not there.
---
## An error with a zero-byte body
The status is right and the body is empty. Some clients treat an empty response as a transport failure and retry, which turns one wrong URL into several requests. A sentence costs nothing.
---
## More than one redirect to reach the homepage
Each hop is a round trip, and agents run with tight budgets: a run capped at a dozen pages spends two of them arriving. The usual chain is http to https to www to trailing slash, four requests where one would do. Collapse them into a single redirect to the canonical origin.
---
## HEAD is refused
`HEAD` answering `405` forces every client that wanted to check a link to download the whole page instead. Link checkers, sitemap validators and agents sampling their own citations all pay for it, and so do you.
---
## No Content-Type on the homepage
Without it a client has to sniff, and sniffing is exactly what `X-Content-Type-Options: nosniff` asks it not to do. Some agents will refuse the response rather than guess.
---
## A Content-Type with no charset
The type is declared and the encoding is not, so a client falls back to a default that may not be yours. This is the mechanism behind mangled accents and broken quotation marks in quoted text.
```
Content-Type: text/html; charset=utf-8
```
---
## Content-Language contradicts html lang
The header says one language and the document says another. An assistant deciding whether your page answers a question asked in Spanish now has two answers, and the one it picks is arbitrary.
---
## 429 without Retry-After
Rate limiting is reasonable. Rate limiting without saying when to come back is not: a well-behaved client has to invent a backoff, and the polite ones wait longer than you needed while the impolite ones do not wait at all.
```
HTTP/1.1 429 Too Many Requests
Retry-After: 60
```
---
## The probe was challenged
A WAF answered the probe with a challenge page rather than the response the origin would have given, so these findings are inconclusive rather than negative. A real crawler may pass the challenge through IP verification or Web Bot Auth where this audit cannot.
> **Inconclusive, not a failure**
>
> The check reports the exact header it saw rather than claiming your site blocks agents. What it does mean is that any fetch-only agent, which is most of them, sees the same challenge.
---
Markdown-Darstellung von https://axrush.com/guides/http-hygiene