Indexability: what head and header say about being indexed
A page that does not stand in a search result can mean three different things: a crawler was not allowed to come read it, a page ban stood in the head, or the page was read but another address was kept as its own. The first question belongs to robots.txt — a crawl rule elsewhere on the host. The second and third are answered by the head of the page itself and the rules of its header, and that is what this article is about.
Indexability reads those two places apart: `meta name="robots"`, directive by directive, with value and any party, and `X-Robots-Tag` in the header, where rules stand that a head may not state. Besides that it reads every `link rel="canonical"` in the head — not only the first — because two blocks with different addresses are a contradiction in the head itself, and every crawler picks one.
What the tool expressly does not do belongs to the story: it does not ask a search engine what it indexed. "No noindex" is a measured absence, not a proven place in an index. This article reads what can be written in a head, which rules contradict each other, and where the measuring stops.
1. Being indexed and passing links on are two questions
One `robots` text holds two kinds of rule mixed together. `noindex` and `none` are about the page itself: may this URL come into the index. `nofollow` is about the links the page hands on: whether a crawler may follow them and what they mean. A page can therefore be indexable with links that are not passed on, or the other way around — and a `none` is no value but a bundle that answers both questions at once.
Then there are the softer rules: `nosnippet` keeps the text out of the result snippet, `noimageindex` keeps images out of the image index, `nocache` and `noarchive` keep stored copies away, and `max-snippet`, `max-image-preview` and `max-video-preview` give measures instead of forbiddances. The last ones can stand beside a party — `max-image-preview:(Google)` or `max-snippet:[Google]300` — and then that party belongs to the directive, not to the value. Whoever only looks for "noindex" misses part of what stands there.
2. Two places where the rule can stand
An index rule can stand in the head, as `meta name="robots"` with a content value, and in the header, as `X-Robots-Tag`. The head is part of the page; the header comes from the server and stands outside the HTML. That difference is practical: a noindex not to be found in the HTML often stands in server configuration, CDN or CMS, and is put there and taken out there. Because a browser may not read the header of another origin, our server fetches the HTML and the headers and the tool reads both in your browser — in one request, at the address of the page itself.
Contradiction between those two places is a finding of its own. If the head says `index, follow` and the header says `noindex`, head and header say opposite things; the tool reports the contradiction and leaves open what a crawler makes of it. The same holds for directives that are not on the fixed list here: an unknown directive is named by name — a typo is possible, something of your own too — and not judged. And an `unavailable_after` with a date that cannot be read is reported as a date a crawler does not read as a date.
3. The address the page states
`rel=canonical` is how a page says which address it means as its own. A crawler keeps that address and gathers there whatever link strength it collects. This tool reads every canonical block in document order: the addresses, whether they come from a relative address or text without a base, whether a fragment (#dst) hangs at it, and whether the address stands on another host or another scheme than the measured URL. Blocks that state different addresses are counted as they are — two addresses in one head are two promises, not one correction.
Next to the canonical often stands `og:url`, the address that sharers keep. If those two differ, one head tells two stories: a crawler keeps the canonical, a sharer keeps the og:url. And next to both stands the address this measurement itself reached — the URL the server asked for after the redirects. Three addresses that are not the same is the most overlooked indexability mistake: the page is indexable, but not at the address its owner meant.
4. Where the measurement stops
A head without noindex proves no index. A head with noindex does prove a ban that stands there — but not that an index saw it standing, nor that an older reading is already gone. Searching is a search engine decision at a moment when you were not present; what is measured here is what stands at this address for the visitor doing this measurement.
The rest of the letter to a crawler also stands with its own tools: robots and sitemap reads what a crawler may come read and crosses that with the site’s own sitemap, Link preview reads what sharers take from the same head, and what Google, LinkedIn and WhatsApp take from your page and robots.txt and sitemap crossed are about the two sides that together decide whether a page is found. A page that wants to be found writes that letter better on purpose than by accident — and the first step is reading what stands there now.