Robots.txt & Sitemap Checker

Make sure search engines can crawl what matters: we read robots.txt, test a URL against Googlebot’s rules and check the XML sitemap it points to.

Googlebot rules
Fast results
100% free
No signup

How it works

How to use the robots.txt & sitemap checker

1
Enter a URL
Use your home page, or a specific page to test whether it’s blocked.
2
We read the rules
robots.txt is parsed with Google’s matching rules and the sitemap is loaded.
3
Unblock & submit
Fix accidental blocks and submit the sitemap in Google Search Console.

What robots.txt does (and doesn’t do)

robots.txt is a plain-text file at the root of your site that tells crawlers which paths they may crawl. It does not remove pages from Google — a blocked URL can still be indexed if other sites link to it. To keep a page out of search results, use a noindex meta tag and leave it crawlable.

How Google reads the rules

Google picks the most specific User-agent group that matches its crawler, then applies the longest matching Allow or Disallow rule; when they tie, Allow wins. * matches any characters and $ anchors the end of the URL. This checker uses the same logic, so you can test any path.

Why your sitemap belongs in robots.txt

An XML sitemap lists the URLs you want indexed, with optional last-modified dates. Adding a Sitemap: line to robots.txt lets Google, Bing and every other search engine discover it automatically — even if you never submit it manually.

FAQ