Crawling internal vs. external URLs is now a setting

We've added a setting that allows you to choose if we should crawl all internal URLs or the external URLs too.

Per-site settings for external URLs

You can now overwrite, per website, if we should probe the outgoing links of your site too.

This is a powerful setting if you want to make sure your visitors are being refered to a working page or website. If your main focus is to aggregate content and refer them to the next website, you'll want to give your users the best experience by linking to working sites.

We do realize this setting isn't for everyone. After all, you have no control over the external sites. They might be down, under maintenance or changing their URL structures.

New default: internal crawling only

Because of the sometimes tricky behaviour of other websites, we've changed the default behaviour to only crawl your sites' internal links, not the external ones.

If you add a new site to your dashboard, we'll stick to the URLs related to the same domain. If you added a site to Oh Dear! before today, you'll notice we're crawling the external links too.

This should give our users a better onboarding experience with less false positives.

Of course, anyone's free to change the setting to also crawl the external URLs!