Splitting Magento Sitemap.xml: Solving the 50K URL Limit
The Google Webmaster Tools limit of 50,000 URLs per sitemap.xml file is a common challenge for large Magento stores. In this post, we’ll address the issue of splitting your sitemap and ensure it works correctly with scheduled cron jobs.
Understanding the Current Issue
You’ve already placed code in app\code\local\Mage\Sitemap\Model\Sitemap.php to split the sitemap if it exceeds 50,000 URLs. This works fine when you manually generate sitemaps from the Magento admin panel.
However, when using a scheduled cron job for automatic generation, the splitting logic isn’t followed properly. The cron job creates two files: sitemap-1.xml with more than 50,000 URLs (causing errors on Google) and sitemap.xml with only a few hundred URLs.
Incorrect Cron Job Behavior
The issue lies in how the cron job is interacting with the Magento Sitemap model. By default, when using the cron job to generate sitemaps, Magento doesn’t honor the custom logic you’ve added for splitting the file based on URL count.
This leads to the unexpected behavior where only a small portion of URLs are included in the main sitemap.xml file while the remaining ones (exceeding 50,000) are placed in an additional sitemap-1.xml file.
Correct Approach for Sitemap Splitting
To ensure your sitemaps split correctly during automated generation via cron jobs, you need to modify the generateXml() method in the same Sitemap.php file where your splitting logic is already present:
public function generateXml()
{
// ... existing code ...
foreach ($this->getUrlCollection() as $url) {
$xml = $this->getXml($url);
$io->streamWrite($xml);
// Insert your splitting logic here
if (count($this->sitemapUrls) % 50000 == 0) {
$this->check_counter($io);
$io->streamClose();
$filename = preg_replace('/\.xml/', '-'.round(count($this->sitemapUrls)/50000).'.xml', $this->getSitemapFilename());
$io->streamOpen($filename);
}
}
// ... remaining code ...
}
In this updated version, we’ve moved the sitemap splitting check inside the loop that processes each URL. This ensures that every time 50,000 URLs are processed, a new file is created.
Conclusion
Remember to clear your Magento cache and re-index all data after implementing these changes. The cron job should now function correctly, creating multiple files with no more than 50,000 URLs in each when generating sitemaps automatically at scheduled intervals.
If you need help optimizing your site’s crawlability or handling large-scale e-commerce SEO challenges like this one, don’t hesitate to contact us for expert Magento development and SEO services.