Guide
Sitemap and robots guide
robots.txt and XML sitemaps are the two files where sites most often quietly sabotage themselves. This guide covers what each file should contain, common mistakes, and how they work together.
01What robots.txt is for
- It instructs well-behaved crawlers which paths they shouldn't request — it is NOT an access-control mechanism.
- Keep it minimal: disallow only what genuinely shouldn't be crawled (admin areas, API endpoints, infinite parameter spaces).
- Never rely on robots.txt to hide private content; anything sensitive must be protected by authentication.
- Blocking a URL in robots.txt prevents crawling but not indexing of its URL — use noindex (crawlable) if a page must leave the index.
02Writing a healthy sitemap
- Include only canonical, indexable URLs that return 200 OK.
- Exclude redirects, soft-404s, duplicates and parameterized variants.
- Keep each sitemap within protocol size limits and split large sites by section.
- Update it automatically when content changes, and reference it from robots.txt.
03Common mistakes
- Disallowing CSS/JS directories, which breaks rendering-based analysis.
- Listing redirected or 404 URLs in the sitemap 'because they used to exist'.
- Conflicting signals: a URL allowed in robots.txt but marked noindex inconsistently across variants.
- Stale lastmod values that make freshness signals meaningless.