Go back
What is the Oh Dear crawler doing in my logs?
If you came to this page, it's probably because you found similar log entries like the ones below in your webserver access logs.
10.0.1.1 - - [28/Nov/2021:08:17:48 +0000] "GET / HTTP/1.1" 200 845 "-" "Mozilla/5.0 (compatible; OhDear/1.1; +https://ohdear.app/checker; brokenLinks)" 10.0.1.1 - - [28/Nov/2021:08:17:48 +0000] "GET /login HTTP/1.1" 200 9155 "-" "Mozilla/5.0 (compatible; OhDear/1.1; +https://ohdear.app/checker; brokenLinks)" 10.0.1.1 - - [28/Nov/2021:08:17:48 +0000] "GET /register HTTP/1.1" 200 11540 "-" "Mozilla/5.0 (compatible; OhDear/1.1; +https://ohdear.app/checker; brokenLinks)" 10.0.1.1 - - [28/Nov/2021:08:17:49 +0000] "GET /password/reset HTTP/1.1" 200 9075 "-" "Mozilla/5.0 (compatible; OhDear/1.1; +https://ohdear.app/checker; brokenLinks)" 10.0.1.1 - - [28/Nov/2021:08:17:49 +0000] "GET /terms HTTP/1.1" 200 8889 "-" "Mozilla/5.0 (compatible; OhDear/1.1; +https://ohdear.app/checker; brokenLinks)"
There are 2 reasons these can show up in your access logs:
- Your website has been added to Oh Dear and we crawl your site for broken links or mixed content errors.
- One of our clients has a website that has outgoing links to your site, and we check to make sure those are still working.
Our crawler is a kind animal that means no harm and will throttle requests if too many are being sent.
Under normal circumstances, this should never hinder or disrupt your site. If it does, please let us know and we will swiftly resolve the issue.
What is the Oh Dear User-Agent?
Our User-Agent is as follows:
"Mozilla/5.0 (compatible; OhDear/1.1; +https://ohdear.app/checker; brokenLinks)"
Or:
"Mozilla/5.0 (compatible; OhDear/1.1; +https://ohdear.app/checker; mixedContent)"
Curious to give Oh Dear a try yourself? Create your account and get started right away!.
