Sitemap Index Files & Splitting
A single XML sitemap is limited to 50,000 URLs or 50MB (uncompressed). For large sites, you must implement a Sitemap Index file that links to multiple sub-sitemaps. Best practice dictates splitting these sub-sitemaps logically—e.g., by category, by publication year, or by product type—rather than arbitrarily. This allows you to monitor indexation rates for specific sections of the site in Google Search Console.
The Importance of Lastmod Accuracy
The <lastmod> tag is one of the few sitemap tags Google heavily relies upon for crawl scheduling. However, if a site generates fake lastmod dates (e.g., setting it to the current day every time the sitemap generates without actual content changes), Google will quickly learn to ignore it. Accuracy is paramount.
Specialized Sitemaps
Beyond standard URLs, large publishers and e-commerce sites require specialized sitemaps to gain rich features:
- Image Sitemaps: Help index images loaded via JS or CSS backgrounds, mapping them explicitly to the host page URL.
- Video Sitemaps: Provide critical metadata (thumbnail, duration, content URL) required for video rich snippets.
- News Sitemaps: Essential for Google News inclusion; must only contain articles published in the last 48 hours.
Dynamic Generation
At scale, sitemaps cannot be static files generated manually or by a scheduled crawler tool. They must be dynamically generated directly from the database or CMS, caching the output for performance but ensuring that deleted URLs are immediately removed and new URLs are appended in real-time.