Robots.txt / Sitemap Checker
Check public crawler-file availability and bounded sitemap discovery.
Run Robots.txt / Sitemap Checker
Public targets only. No credentials, changes or anonymous active security testing.
What this check does
- Public robots.txt response
- Sitemap discovery within the supported boundary
- Bounded response metadata
What it does not prove
- No full sitemap URL crawl
- No indexing or ranking guarantee
- No assessment of all content quality
Interpret the returned state
PASS means the endpoint returned evidence within its named boundary. REVIEW, WARN and FAIL must be read alongside that evidence; they are not automatically vulnerabilities. UNKNOWN means the observation could not establish the answer.
Why it matters
Crawler files communicate discovery and crawling instructions. Their existence does not prove Google indexed a page or that a blocked page is confidential.
Common review questions
- robots.txt or a sitemap may be missing or routed incorrectly.
- A published sitemap can still contain redirects or stale URLs.
- Robots directives are not an access-control mechanism.
Separate crawling, discovery and indexing
Start by checking the public files served for the intended hostname. A working homepage does not prove the crawler files are routed correctly.
Read the bounded response and discovery evidence. This endpoint does not fetch and verify every listed page.
Compare sitemap canonicals and HTTP outcomes in a separate authorised route audit. Search-engine indexing is a further decision and is not guaranteed by this tool.
When a fuller review would help
ZENQIX can combine relevant observations, explain priorities and recommend next steps. No automatic remediation, penetration test or complete-security guarantee is included.
Explore Website AssuranceFrom ₹4,999