What is the Oh Dear! crawler doing in my logs?

If you came to this page, it's probably because you found similar log entries like the ones below in your webserver access logs.

10.0.1.1 [12/Jun/2018:11:33:00 +0200] "GET / HTTP/1.0" 200 83890 "-" "Mozilla/5.0 (OhDearApp; compatible)"
10.0.1.1 [12/Jun/2018:11:38:23 +0200] "GET /robots.txt HTTP/1.0" 302 213 "-" "Mozilla/5.0 (OhDearApp; compatible)"
10.0.1.1 [12/Jun/2018:11:38:27 +0200] "GET /docs/api/authentication HTTP/1.0" 302 226 "-" "Mozilla/5.0 (OhDearApp; compatible)"
10.0.1.1 [12/Jun/2018:11:38:27 +0200] "GET /deeply/nested/page HTTP/1.0" 200 24 "-" "Mozilla/5.0 (OhDearApp; compatible)"
...

There are 2 reasons these can show up in your access logs:

  1. Your website has been added to Oh Dear! and we crawl your site for broken links or mixed content errors.
  2. One of our clients has a website that has outgoing links to your site, and we check to make sure those are still working.

Our crawler is a kind animal that means no harm and will throttle requests if too many are being sent.

Under normal circumstances, this should never hinder or disrupt your site. If it does, please let us know and we will swiftly resolve the issue.

Curious to give Oh Dear! a try yourself? Create your account and get started right away!.