What is the Oh Dear crawler doing in my logs?

If you came to this page, it's probably because you found similar log entries like the ones below in your webserver access logs. [12/Jun/2018:11:33:00 +0200] "GET / HTTP/1.0" 200 83890 "-" "Mozilla/5.0 (compatible; OhDear/1.0; +https://ohdear.app/crawler; Run 12345)" [12/Jun/2018:11:38:23 +0200] "GET /robots.txt HTTP/1.0" 302 213 "-" "Mozilla/5.0 (compatible; OhDear/1.0; +https://ohdear.app/crawler; Run 12345)" [12/Jun/2018:11:38:27 +0200] "GET /docs/api/authentication HTTP/1.0" 302 226 "-" "Mozilla/5.0 (compatible; OhDear/1.0; +https://ohdear.app/crawler; Run 12345)" [12/Jun/2018:11:38:27 +0200] "GET /deeply/nested/page HTTP/1.0" 200 24 "-" "Mozilla/5.0 (compatible; OhDear/1.0; +https://ohdear.app/crawler; Run 12345)"

There are 2 reasons these can show up in your access logs:

  1. Your website has been added to Oh Dear and we crawl your site for broken links or mixed content errors.
  2. One of our clients has a website that has outgoing links to your site, and we check to make sure those are still working.

Our crawler is a kind animal that means no harm and will throttle requests if too many are being sent.

Under normal circumstances, this should never hinder or disrupt your site. If it does, please let us know and we will swiftly resolve the issue.

What is the Oh Dear User-Agent? #

Our User-Agent is as follows:

"Mozilla/5.0 (compatible; OhDear/1.0; +https://ohdear.app/crawler; Run 1181035128)"

The fixed part of the crawler is this:

"Mozilla/5.0 (compatible; OhDear/1.0; +https://ohdear.app/crawler; Run

The value after Run is used to identify the current check being performed, which can help you correlate broken links or server-side issues with our reports.

Curious to give Oh Dear a try yourself? Create your account and get started right away!.

Was this page helpful to you? Feel free to reach out via support@ohdear.app or on Twitter via @OhDearApp if you have any other questions. We'd love to help!