Cross-checking robots.txt and sitemap
robots.txt is no wall — it is a request, standardised in RFC 9307, and crawlers may follow or ignore it. sitemap.xml is no indexing order — it is a list you offer. The fault sits not in either one apart, but in the combination: URLs that you block in robots.txt and offer in your sitemap send crawlers a contradiction.
The robots & sitemap tool fetches both documents through our server and lays them side by side: rule by rule what you forbid and allow, and the cross-list of URLs that robots blocks while the sitemap names them. A sitemap that does not arrive as XML, or a Disallow on the wrong path, then simply stands in the report.
This article explains what each rule means, which combinations jar, and why "noindex" is something other than "Disallow" — with the tools that can measure the difference.
What robots.txt is and is not
A Disallow is a request not to crawl; it does not technically block a crawler and does not hide a URL. If you want a page not to enter the index, the means is the noindex meta or the HTTP header — and those can only work if the crawler may read the page. That is exactly where the combination with robots.txt jars: a blocked page whose noindex nobody could read sometimes simply stays in the index.
The cross-check that counts
The most common contradiction: a sitemap offering URLs that robots.txt just forbade. Crawlers do not treat that uniformly — some ignore the sitemap entry, some register the contradiction as a signal of sloppy management. The tool shows the cross-list; what you do with it (withdraw from the sitemap, or lift the stay in robots) is a policy choice you make with the list before your nose.
Rules that often work just differently than thought
User-agent-specific rules hold per crawler — what holds for Google-Image need not hold for an ordinary crawler. A `Disallow: /*?` with wildcard does not work the same everywhere; RFC 9307 defines what may stand, and the rest is the reader's interpretation. Crawl-delay is ignored by the big houses. The tool judges the text as RFC 9307 defines it and leaves doubt as doubt — it is your document, not our verdict.
Where this leads
A site that presents itself well to crawlers also presents itself well to sharers: an OG-image that robots.txt blocks, a sharer does not see — what Google, LinkedIn and WhatsApp take from your page is that story from the other side, and securing a website begins where these articles end: measuring what stands, not guessing what stands.