Oh Dear crawls websites to report broken links and mixed content reporting. You can configure which pages we can and cannot crawl by adding us to your robots.txt file.
robots.txt files look similar to this.
User-agent: * Disallow:
This essentially instructs any crawler (that adheres to the robots.txt spec) to crawl every page.
To limit the crawls, you can add our User-Agent to the list.
User-agent: OhDear Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /junk/
This instructs the Oh Dear crawler to ignore the