Question 1

Where does this tool look for sitemaps?

Accepted Answer

First, the Sitemap: directives in robots.txt. Then it tries the standard fallbacks at /sitemap.xml and /sitemap_index.xml. Sitemap indexes are followed one level deep so you can see child sitemap URL counts.

Question 2

Why is my Disallow rule still allowing Googlebot?

Accepted Answer

robots.txt only blocks crawling, not indexing. If a URL is linked from elsewhere it can still appear in Google search with no snippet. Use a noindex meta tag (or X-Robots-Tag header) to remove it from the index.

Question 3

Can I check if a specific URL is allowed for Googlebot?

Accepted Answer

This tool shows the rules — to test a URL against them, paste both into Google Search Console's robots.txt Tester. The Pyrelo view is faster for spotting bad Disallow patterns at a glance.

Question 4

What does Allow: / mean if there's no Disallow?

Accepted Answer

It's a no-op — by default, everything is allowed. Authors sometimes add Allow: / for clarity, especially in conjunction with bot-specific User-agent: blocks. The crawler treats absent rules the same as Allow.

Robots.txt & Sitemap Inspector

More Website Tools

HTTP Headers Inspector

SPF / DMARC / DKIM Checker

Page Weight Analyzer

The all-in-one work dashboard for small teams