Using Oh Dear! to keep your Varnish cache warm

Slow websites are annoying, right? We sure think so.

One common solution is to introduce a caching proxy like Varnish to help cache pages and reduce your server load.

The good news is, if you have Oh Dear!, you can let those 2 work together.

Varnish + Oh Dear! crawling = ❤️

The idea is as follows: if you've enabled our broken links or mixed content checks for any of your sites, we'll crawl your sites to find any broken pages.

On top of that, we have the ability to set custom HTTP headers per website that get added to both the uptime checks and our crawler.

Combining our crawler and the custom HTTP headers allows you to authorize our crawler in your Varnish configs to let it update the cache.

Configuring Varnish to refresh your cache when Oh Dear! crawls your site

There are several default Varnish configs available you can use as the boilerplate for your Varnish service. One of them is maintained by one of the co-founders of Oh Dear! and can be found here: Varnish 6 configuration template.

If you take this as your basis, it's relatively easy to allow Oh Dear! to update your cached pages.

First, add this line to your vcl_recv routine.

vcl_recv {
  if (req.http.OhDear-Authorized-Request == "ASecretStringOrPassphrase") {
    set req.hash_always_miss = true;
  }
  
  ...
}

Now, every time Varnish receives a request with the header OhDear-Authorized-Request that has the value ASecretStringOrPassphrase, it will refresh the current page from its backend (aka: you won't be served a cached response, but it will go to your backend server).

In Oh Dear!, you can add this custom header to each of your sites.

From now, that extra header will be present every time we crawl your page.

This also allows extra possibilities in your Varnish code, like authorizing our crawler to access otherwise limited pages.

Checking the uptime behind your cache

Using similar logic, we can also allow the uptime checks to bypass the cache altogether to check the availability of your site.

It wouldn't be the first time a site appears online, because the monitoring probes were answered by a cached response. Your users that bypass the cache (they might be logged in) are still seeing your error pages.

To let Oh Dear! bypass the cache every time without touching the TTL (Time-To-Live) or currently cached pages, you can use logic similar to this.

vcl_recv {
  if (req.http.OhDear-Authorized-Request == "ASecretStringOrPassphrase") {
    return (pass);
  }
}

We're certain there are other use cases out there that we haven't even thought of. If you're doing something similar, we would love to hear from you in the comments below!