Sitemap monitoring

A sitemap is a file where you can list the web pages of your site to tell Google and other search engines about the organization of your site content. Search engine web crawlers like Googlebot read this file to more intelligently crawl your site.

We can monitor your sitemap by checking if it's available and if the structure is valid. We will also visit each URL in the sitemap to check if it's available and if it returns a response code in the 200 or 300 range.

Sitemap indexes

If you have a very large sitemap, you could opt to split it up in multiple ones and link to them using a sitemap index file. Our sitemap check also support such an index file. It will check all of sitemaps mentioned in the index file.

Gzipped sitemaps

We also support sitemaps that are gzipped. If your site map URL ends with .gz, we will automatically decompress it and check the sitemap.

Specifying the location of your sitemap

When you add a new site to Oh Dear, you can choose to activate sitemap monitoring. By default, we'll try to find your sitemap at the default location /sitemap.xml

You can also specify a custom location for your sitemap in the settings tab of the sitemap check of your site.

Sitemap checks

We will check the following things when monitoring your sitemap. We will check if:

• the sitemap is available at the specified URL
• visit each URL in the sitemap to check if it's available and if it returns a response code in the 200 or 300 range
• all URLs are unique
• the sitemap doesn't contain more than 50,000 URLs
• the priority of each item is between 0 and 1
• the frequency of each item is one of these values: always, hourly, daily, weekly, monthly, yearly, never
• last modified of each item is not in the future

If you are using a sitemap index, we will perform all these checks for all sitemaps mentioned in the index file.

Limits

To protect our infrastructure, we've imposed these limits:

• we will only check the first 50 sitemaps in a sitemap index file
• we will check the structure of each sitemap, but we will visit only the first 5000 URLs
• we'll spend a maximum of 15 minutes visiting URLs, after that we won't check the reachability of URLs anymore

We think the vast majority of users won't hit these limits. If you do, get in touch with us, and we'll see what we can do.