robots.txt & Sitemap Checker
Fetches robots.txt and the sitemap, tests whether a URL is crawlable and validates the sitemap format.
Fetches a site’s robots.txt and sitemap, then rules on whether a given path is crawlable — per crawler, showing which group and which rule decided it. It answers "why is this page not indexed" at the one layer that silently blocks everything else.
Our server makes this request for you. Only the URL you typed is sent to it — none of your cookies or login state goes along. The server will only connect to public addresses and refuses private and internal ones.
How to use it
Paste the page URL, not the domain
The path is what gets judged. robots.txt is fetched from that URL’s host automatically.
Read the verdict per crawler
Googlebot, Bingbot, the AI crawlers and any User-agent you add are each judged separately, because robots.txt groups them separately. The deciding rule is shown with its line number.
Check the sitemap alongside it
The sitemap is fetched from the Sitemap: line, or /sitemap.xml if there is none, and its entries and problems are listed. A sitemap that returns 404 is what Search Console reports as unfetchable.
Common questions
Does robots.txt keep a page out of search results?
No — it stops the crawl, not the listing. A blocked URL that other sites link to can still appear, without a description, because the crawler is forbidden from reading the noindex you put on it. To remove a page, allow the crawl and serve noindex.
My robots.txt returns a 500 — does that matter?
A great deal. Under RFC 9309 a 5xx means the file is unavailable, and a crawler treats the whole site as disallowed rather than assuming the best. A server error on this one file can take a site out of search entirely.
Two rules match my path — which one wins?
The longest match, and Allow wins a tie (RFC 9309 §2.2.2). The table names the rule that actually decided, which is usually the surprise.
Do crawlers actually obey this?
The named ones do. robots.txt is a convention, not an access control — anything ignoring it will fetch the page regardless, so a path that must not be public needs authentication rather than a Disallow line.
Feed it from
Link Preview Debugger
Reads Open Graph and Twitter card tags from a URL and renders how the link will look when shared.
Broken Link Checker
Extracts up to 100 links from a page, or takes a pasted list, and checks each in parallel — status, redirect target and response time, HEAD first and GET when refused.
WHOIS Lookup
Shows domain registration, registrar, nameservers and expiry, plus RDAP data where it exists.
QR & Barcode Reader
Reads QR codes and common barcodes from an uploaded image, a screenshot or the camera.
Next
These accept what this tool produces, so the result can go straight in.
HTTP Header Inspector
Fetches a URL and shows the full response headers, redirect chain, timings and compression used.
Broken Link Checker
Extracts up to 100 links from a page, or takes a pasted list, and checks each in parallel — status, redirect target and response time, HEAD first and GET when refused.
Link Preview Debugger
Reads Open Graph and Twitter card tags from a URL and renders how the link will look when shared.
HTTP Request Builder
Composes and sends a GET/POST/PUT/PATCH/DELETE/HEAD request with custom headers and a body, and shows the status, headers, body and timing — without ever forwarding cookies or Authorization.
Barcode Generator
Generates EAN-13, UPC, Code128, Code39 and ITF barcodes with check digits and print-ready output.
CSV Cleaner
Trims cells, normalises headers, fixes number and date formats and reports rows with the wrong column count.