It's called the ghost domain problem. It's documented in DNS research circles and has an active IETF draft (draft-ietf-dnsop-ns-revalidation), but it's niche enough that an uptime product can run for years without anyone noticing it. This post is about why, and the small set of changes we're making on our end.

How it shows up #

The .de zone has a strict holder-data verification policy. If the registrant's details aren't verified within DENIC's deadline, DENIC first removes the domain from the .de zone and deletes it later. And .de is far from alone. EURid suspends a .eu domain and, if the data still isn't verified, moves it to a withdrawn (no longer registered) state. AFNIC runs an eligibility and contactability check on .fr holders and can suspend, then eventually delete, a domain whose holder doesn't correct or substantiate their data.

And across ICANN gTLDs like .com, .net, and .org, the 2013 WHOIS Accuracy Program requires the registrar (not the registry) to suspend, terminate, or place a domain on clientHold after more than 15 days without a response to a verification request, which pulls it from zone publication. Failed verification, expired registration, registrar disputes, abuse takedowns: the names differ, the shape doesn't. The registry or registrar pulls the name from the parent zone, but the domain's own authoritative nameservers keep answering as if nothing changed.

For anyone whose resolver no longer holds the child-side records, that means NXDOMAIN (RCODE 3, "name does not exist"). A fat error in the address bar, no site. For an uptime checker sitting behind a long-lived recursive resolver cache that does still hold them, it can mean nothing at all. The difference isn't browser versus checker, it's a cold cache versus a warm one. The site looks fine. It isn't.

Where the bug lives #

This was the DNS chain we had on one of our uptime workers (Frankfurt, in this case):

PHP/curl
  → glibc nsswitch (files dns)
  → /etc/resolv.conf (127.0.0.53)
  → systemd-resolved stub
  → upstream recursive resolver (shared, run by the hosting provider)
  → authoritative nameservers (the domain's DNS provider)

I'll spare you the full diagnostic. The summary: systemd-resolved is a local caching stub. The real resolution work happens in a recursive resolver upstream, and that's where the bug lives. Not on our box, and not on yours. This is inherent to how recursive DNS works, no matter who runs the resolver: the caching layer that makes DNS fast is the same layer that can keep a dead delegation warm.

That detail matters. We could ship the most diligent uptime checker on the planet and still get the wrong answer, because we're trusting a cache we don't control.

Ghost domains, briefly #

The mechanism is documented in academic literature (the NDSS Ghost Domain Names paper from 2012, and a 2023 follow-up Ghost Domain Reloaded). It currently sits in an active IETF draft, Delegation Revalidation by DNS Resolvers. The credibility rule it depends on is in RFC 2181, and the delegation-versus-glue terminology in RFC 9499 (which obsoleted RFC 8499).

Here's how it plays out, step by step:

Step 1: The first lookup

The first time a recursive resolver looks up example.de, it walks down to the .de (parent) servers, which hand back a referral: "I don't host example.de, but its nameservers are ns1.example-dns.net and ns2.example-dns.net." That's the parent-side delegation NS RRset, the NS records for example.de, as published by .de.

The resolver then asks those nameservers directly, and their authoritative answer usually carries the same two-name list again, this time published by example.de itself from its own zone apex. That copy is the apex NS RRset. Auth servers aren't required to include it, but they usually do, and the ghost effect depends on it being cached.

Step 2: The child's copy outranks the parent's

Both RRsets name the same two servers; the difference is who said so. Per RFC 2181's ranking rules, the child's authoritative apex copy (from example.de) outranks the parent's referral copy (from .de), so it replaces the parent-side one in cache. To be clear, this is the NS RRset for example.de itself, not the NS records of the .de zone.

Step 3: The A record expires

Later, when the A record expires, the resolver asks "where do I find example.de?", finds the cached apex NS RRset (still pointing at ns1 and ns2.example-dns.net), and queries those nameservers directly.

Step 4: The authoritative servers still answer

The authoritative nameservers still answer positively. They have no idea the registry pulled the delegation. Why would they?

Step 5: The loop

A fresh A record gets cached. Another TTL window starts. Loop.

Absent any revalidation, the parent zone is ordinarily not consulted again as long as the apex NS RRset stays warm. And the defaults don't help:

It's tempting to assume frequent uptime checks are what keep the cache hot. Not quite. What keeps a ghost alive is the resolver reacquiring the apex NS RRset before it ages out, and how long that RRset lives is bounded by its own TTL and the resolver's cap. Check frequency only matters as opportunities to reacquire it, so cutting back on checks won't save you: a domain with a long NS TTL can keep ghosting off very few lookups per cache window.

What every other monitoring service does #

I went looking. I read the public docs, FAQs, and architecture pages of Pingdom, UptimeRobot, StatusCake, Site24x7, Datadog Synthetics, and Better Stack.

I couldn't find a documented defense against this in any of their public materials.

Pingdom actually documents the architecture that causes the bug, in their own blog post:

Each of Pingdom's probe servers run their own individual Bind9 caching DNS server as their DNS resolver, so DNS records will be cached.

We don't know how they've tuned it, and a per-probe BIND is a perfectly reasonable setup. But BIND's default max-cache-ttl is one week, and unless that's been lowered, it's exactly the kind of cache horizon where a ghost domain can stick around for days.

A few of these tools monitor domain expiry, and so do we: Oh Dear has domain expiration monitoring that warns you well before a renewal date slips past. But that's adjacent, not the same thing. Expiry monitoring watches the registration date, not whether a still-paid-for domain has been pulled from the zone over a failed verification. I couldn't find any tool, ours included, that treats registry-level delegation removal as a distinct case in its uptime checks.

This isn't a gotcha against any of them. It stayed off our radar for the same reason it probably has for theirs: it's a DNS-layer problem hiding under an HTTP-layer product, invisible until someone goes looking.

What we're tightening #

We're keeping this narrow.

A local recursive resolver on each checker, with a tight cache. We're running Unbound on each worker, doing full recursion itself instead of forwarding to a shared upstream resolver. The point isn't "ours is better." It's that we get to set the knobs.

Two of them matter here:

Unbound caps cached records at a day by default; we clamp that to an hour. Once the cached apex NS RRset expires, the next lookup has to restart iteration from the closest cached cut, typically the TLD's nameservers, to find the delegation again, and that's exactly where a pulled delegation now comes up empty. harden-referral-path (paired with a deeper target-fetch-policy, as the Unbound docs recommend) makes the resolver re-check the NS data it meets on the way down instead of trusting stale child-side state.

That's the main change. It doesn't make ghost domains impossible, but it shortens the window in which a stale delegation can lie to us from days down to about an hour.

We've thought about heavier options. Cross-resolver verification on failed checks, where we'd cross-check with 1.1.1.1, 8.8.8.8, and 9.9.9.9 in parallel. Dedicated parent-zone delegation polling per monitored domain, à la dig @f.nic.de example.de NS. Both are real fixes, and we may come back to them. But the operational cost (extra DNS load, public resolver rate limits) is hard to justify when tightening our own resolver already covers the realistic cases.

What this doesn't fix #

DNS caches exist for good reasons. The whole reason a recursive resolver works the way it does is to keep the global DNS from melting every time someone reloads a page. We're not going to "fix DNS." What we can do is narrow the window where a stale delegation goes unnoticed.

If a ghost domain's apex NS RRset keeps refreshing inside our shorter cache window, it can still slip past us. Other layers (the registry's own propagation cadence, the authoritative DNS host, the resolvers downstream from us) all behave their own way. We're reducing surface area, not eliminating it.

And we're honest about the tools we reached for. harden-referral-path is experimental and not an RFC standard, so we run DNSSEC validation alongside it in permissive, log-only mode: bogus answers get logged for us to review, not turned into SERVFAIL, so a signing glitch on a customer's domain can't show up as false downtime. That's a deliberate first step. Once the logs stay clean and the rollout proves stable, we plan to flip DNSSEC to enforcing over the coming weeks.

What you should do #

If you run anything important on a domain you don't own at the registrar level, run DNS monitoring alongside uptime monitoring. Uptime monitoring answers "can my site be reached?" DNS monitoring answers "is the DNS layer healthy?" Together they catch problems neither one would on its own.

In Oh Dear, you turn DNS monitoring on per monitor in the DNS check settings. It watches your A, AAAA, MX, NS, and TXT records and alerts when they drift. It's part of your plan, not a paid add-on: every monitoring check we offer is included, at no extra cost.

If you're not an Oh Dear customer yet, you can try it for 10 days, free, no credit card needed. And if you hit a monitoring edge case we don't catch yet, show us. We'll make Oh Dear better by building the fix in.

We're working with several partners to bring Oh Dear into more products and platforms. Zone and Ploi Cloud are the first two out the door.

What's the reseller program? #

If you run a hosting company, a deployment platform, or an agency managing client sites, you probably want monitoring baked into your product. Building that from scratch is a massive undertaking. Uptime checks from multiple global locations, SSL monitoring, broken link detection, performance tracking, DNS change alerts, cron monitoring... that's years of work.

Our reseller program lets you skip all of that. You integrate with our API, and your customers get the full Oh Dear experience inside your platform. You handle the billing and the relationship with your customer. We handle the monitoring infrastructure.

Zone: monitoring as a native hosting feature #

Zone is one of the largest domain registration and hosting providers in the Baltics, operating in Estonia, Finland and Latvia. They now offer Oh Dear monitoring as a native service to all their customers, covering uptime, SSL certificates, broken links, performance, domain expiry, DNS changes, and cron job monitoring.

Their approach to notifications is seamless: Zone customers get a one-click login into the full Oh Dear dashboard, where they can configure exactly which alerts they want and where they should go. Email, SMS, Slack, webhooks, you name it. The customer controls their own notification preferences directly in Oh Dear, without ever needing to create a separate account.

What we love about Zone's integration: they don't just resell it, they use Oh Dear internally too. When your own hosting provider trusts the same monitoring tool they're offering you, that says something.

"Oh Dear is the easy one to implement, because the API makes sense and is easy to work with."

Kaarel Urva, Chief Project Officer at Zone Media

You can read more about their offering in Zone's announcement post.

Ploi Cloud: three clicks to monitoring #

Ploi Cloud is a deployment platform built for developers. Their Oh Dear integration is beautifully simple: navigate to the Monitoring tab, click "Enable monitoring", and you're done. Uptime checks, SSL monitoring, performance metrics, broken link detection, and mixed content scanning, all running within seconds.

Ploi took a different approach to notifications. Instead of sending users to the Oh Dear dashboard, they receive all monitoring alerts through our webhooks and pipe them into their existing notification system. If you've already set up Slack, Discord, Google Chat, or email notifications in Ploi, your monitoring alerts just show up there too. Nothing extra to configure.

Their team described it well: "This isn't a third-party plugin or a workaround, it's a first-class integration built together."

Check out Ploi Cloud's announcement for the full details.

Built on our reseller API #

Both integrations are powered by our reseller API. It gives partners everything they need to build a seamless experience:

• Team provisioning: create and manage separate client teams programmatically
• User management: add users with roles (admin, member, guest) to each team
• Secure login links: generate time-limited login links so customers can access the full Oh Dear dashboard without creating separate credentials
• Monitor management: create, configure, and manage monitors across all client teams through the standard monitors API

The beauty is in the flexibility. Zone gives their customers direct access to the Oh Dear dashboard through login links, so users can configure their own notification preferences. Ploi keeps everything inside their own platform and uses webhooks to feed alerts into their existing notification channels. Both approaches work, both are valid. You tailor it to what fits your product best.

More to come #

Zone and Ploi Cloud are the first two reseller partners we can publicly share, but there are more in the making. We'll announce them as they go live.

If you run a hosting company, a deployment platform, or a service where website monitoring would be a natural fit for your customers, we'd love to talk. Reach out at support@ohdear.app and we'll walk you through the program.

Oh Dear is an all-in-one monitoring service that gives you a unified dashboard for uptime checks, performance monitoring, broken link detection, SSL and domain expiry alerts, scheduled task validation and more. But beyond just running checks Oh Dear also gives you powerful tools for organizing and automating how you manage your client sites at scale.

Here are 5 practical ways to make that happen.

1. Use Tags to Organize and Notify #

When you're managing a fleet of sites, grouping them by client, environment, region or team is essential. Oh Dear lets you apply tags in any format but we suggest something like:

• client:spatie
• env:production
• infra:cloudflare
• region:eu-west-1
• project:rebrand

You are probably already aware that you can route notifications to tags or groups of tags. Once your tags are set up, you can configure tag group notifications. This means when something goes wrong on any site tagged with env:production + client:* for example, you can notify a VIP Slack channel or OpsGenie team automatically - no manual routing needed while env:staging + internal can go to the email inbox that no one reads.

See it in practice: how full-service agency Marbles uses Oh Dear.

Tags also make onboarding easy. New production application for a client? Just tag it accordingly client:spatie and all the right checks and notifications apply.

2. Find Sites by Tag via Global Search #

Tags aren't just for routing notifications - they’re fully searchable. Open the global search menu in Oh Dear and type any tag (like region:eu-west-1 or team:seo) to instantly filter your sites.

This is especially handy for large accounts where you need to quickly check in on just the production sites in Europe or just the ones belonging to a specific customer.

3. Trigger Health Checks on Deployment via the API #

Oh Dear provides an API endpoint to trigger checks manually, which is perfect for CI/CD workflows.

This ensures you catch issues before the client does, and keeps your deployment pipeline tightly integrated with live monitoring. Here's how to request a new run of any check, for example the broken links check: https://ohdear.app/docs/api/checks#request-a-run

But you can also apply the same principles to all of our other checks. My favourites are:

• Lighthouse
• Sitemap

const OHDEAR_TOKEN = 'your API token';

// you can find this via the /sites/{id} endpoint
// or from the check settings in the app
const checkId = 1; 

fetch('https://ohdear.app/api/checks/1/request-run', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${OHDEAR_TOKEN}`,
    'Accept': 'application/json',
    'Content-Type': 'application/json'
  }
})
  .then(res => {
    if (!res.ok) throw new Error(`HTTP ${res.status}: ${res.statusText}`);
    return res.json();
  })
  .then(data => console.log('Check run requested:', data))
  .catch(err => console.error('Error triggering check:', err));

Our uptime checks and application health checks run every minute - so that's two extra things we don't have to worry about.

You could also add a maintenance window if you don't have zero-downtime deployments or you are expecting some temporary latency after go-live. Read more here: https://ohdear.app/docs/features/how-to-configure-maintenance-windows

4. Automate Site Creation Using Templates #

When spinning up new client sites speed and consistency is key. Rather than configuring everything from scratch, you can:

• Assign one of your sites as a template site using tag: template with your base checks, tags and settings.
• Use the API to clone that site when onboarding a new client.
• We support all site settings from our API so you can customize the new site as needed (e.g. domain name, team assignments) at the same time.
• Add the site to an internal status page and client-facing status page both with custom domains. (Yes you can have as many status pages, custom domains and sites as you need. Included in every plan!)

This approach saves time, ensures consistency across all clients and reduces configuration drift.

You are in good company. Oh Dear is the preferred tool for Dutch education and research.

5. Control Access with Fine-Grained Permissions #

Not every stakeholder needs to see everything. Oh Dear lets you invite clients or teammates with scoped access:

• Give dev teams access to performance and scheduled task checks.
• Let clients see uptime and SSL status only.
• Prevent accidental edits to monitoring settings.

This keeps your workspace clean and secure while still being transparent with stakeholders.

You can also add clients to the monthly site reports so they will automatically be up to date. We run reports on the first day of every month.

Tip: client email address can be added to the monthly site report via sites API.

Bonus: Add Site Status Badges #

Want to show the latest status on your client sites? Oh Dear provides a site badge you can embed showing current uptime status.

// https://ohdear.app/api/sites/{id}

{
   ...
   "badge_id": "01jeb77smcek1f4dxys08vect5",
   ...
}

Then update your content with:

<img src="https://ohdear.app/badges/site/{badge_id}/uptime/normal" alt="uptime status badge" />

It's a small touch that builds trust with clients and surfaces real-time updates to visitors.

What's Next? #

In the next few blog posts, we'll dive deeper into each of these tips with hands-on code examples using the Oh Dear API, tips for setting up automation and ideas for streamlining your agency workflows even further.

Stay tuned!

Leading causes of website downtime #

Before we jump into ways of preventing website downtime, let’s first cover the leading causes of website downtime. If you can diagnose the cause of the downtime, you’ll be in a much better spot to prevent it from happening again. You don't want the perpetual Whac-a-Mole. So, here are the eight leading causes of website downtime:

• Server issues: Whether you run your own website servers or pay for third-party hosting, hardware or software issues with your website’s servers can cause your site to go down.
• Overwhelmed resources: A sudden spike in traffic to your website can cause your servers to become overwhelmed preventing additional traffic from accessing your site.
• Expired SSL certificates: Although this doesn’t cause your website to go down, an expired SSL certificate should be treated the same. Most users will navigate away from your site if they receive a warning saying your site is insecure which occurs with an expired SSL certificate.
• DNS configuration issues: Your DNS needs to be configured correctly so that users are directed to your site when they navigate to your site’s URL. A DNS issue prevents people from accessing your site through your domain name.
• Software conflicts: Most sites use software to handle the backend of their website. Any errors with the software, which can occur after an update or due to conflicts between multiple software running different parts of a site’s backend, can cause your site to go down.
• Cyberattacks: Malicious actors can bring your website down with cyberattacks such as DDoS attacks which artificially overwhelm your hosting infrastructure.
• Human error: Mistakes happen more often than we think. Thankfully, if it was human error that took down your website, it usually means you can get it back up by reverting the action that caused the downtime.
• Maintenance gone awry: During scheduled maintenance, things can go wrong that cause the downtime to extend beyond the planned maintenance time.

10 ways to help you to prevent website downtime #

So, how do you prevent those eight leading causes of website downtime from occurring? We’re going to cover the 10 most effective strategies to prevent website downtime and minimize its impact.

Implement global uptime monitoring

To prevent website downtime, you first need to know when it happens. That can help you identify issues causing your website to go down. It also allows you to take immediate action to bring your site back up when it goes down.

If you are new to this, start with how website monitoring actually works.

Monitor your SSL certificates

Like we mentioned, issues with your SSL certificate won’t cause website downtime, but it does divert traffic. When issues with your SSL certificate exist, anyone navigating to your website will be greeted with a big warning from their browser saying your site is not secure. Most people do not choose to proceed to your site after that warning. SSL certificate monitoring ensures you’re the first person to know about any issues with your certificate so you can fix it before they impact your users.

Set up performance Monitoring

Pages that load slowly can have just as big of an impact as a page being down. When a page takes over three seconds to load, nearly half of users will leave your site. Performance monitoring keeps an eye on how long it takes your site to load and tells you what’s bogging down your load times. That gives you the information you need to make your site faster.

Implement DNS monitoring

Internal tests can easily miss DNS errors. Afterall, an issue with your DNS won’t prevent your server from sending data to a test environment. But, it does prevent your users from accessing your website. You need your DNS configured correctly to ensure your users can access your site through your domain name. DNS monitoring checks your configuration, ensures users can access your domain, and tells you where the error exists if there is an issue with your DNS records.

Schedule regular backups

Downtime is not 100% avoidable. There will be errors that happen whether they’re human errors, an issue with your hosting servers, or a software issue. Scheduling regular backups allows you to rollback your changes at any time while minimizing progress loss. This can immediately bring your website back up if you run into software issues or a human error that takes down your site.

Implement load balancing

Without load balancing, a single server becoming overwhelmed can prevent some users from accessing your website. Load balancing spreads out the traffic to your website between multiple servers preventing any one server from becoming overwhelmed. This makes your website more scalable as you can add more servers to accommodate increases in traffic.

Monitor scheduled tasks

Whether its database or server maintenance, keep track of scheduled tasks and monitor their progress. These tasks can fail and if not monitored, can cause cascading effects that create downtime for your website.

Check for mixed content

Mixed content doesn't cause downtime, but it does create a poor user experience. Browsers can identify mixed content and then send users a security warning just like with SSL certificate issues. Monitoring for this type of content tells you what pages it affects so that you can fix it.

Set Up application health monitoring

Most modern websites use various applications to run all the aspects of their site. These applications need to work in tandem for every part of your site to work. Application health monitoring notifies you of issues within an application and with the integration of each application so that it can work cohesively.

Create and maintain a status page

When downtime occurs, communication is critical. You want your users to know that your website will be coming back up and, if you can, give them a timeframe of when they can expect your site to be back up. So, create and maintain a status page with all relevant information you can display during downtime.

Monitoring your website #

One more: do not forget your cron jobs, scheduled task monitoring covers those.

It can sound overwhelming to manage all this stuff just to prevent downtime and fix issues before they impact your users. But guess what, modern website monitoring solutions—like Oh Dear of course—can manage all of the above with minimal effort from your end. They offer you a single dashboard to track all relevant aspects of your website from your performance and uptime to your SSL certificate and DNS records. You can try OhDear for free for 10 days to see how website monitoring can help you better manage your sites.

FAQs about website downtime #

How can I quickly respond to website downtime? Website monitoring is the best way to get notified of website downtime immediately so that you can respond to it as quickly as possible.

Should I notify users about downtime? Yes, proactive communication is the best way to minimize the impacts of downtime to your end user. Be open about the cause of the downtime, how long you expect it to impact users, and the extent of the impact.

How do I diagnose the cause of website downtime? You can start your diagnosis with the monitoring tool that notified you of the downtime. It should tell you the general cause of the error based on the HTTP status code the monitoring service received to identify the downtime. From there, you need to systematically test ever aspect related to the general cause until you find the point of failure.

What should I include in my website downtime status page? When displaying a status page notifying users that your website is currently down, we recommend including information related to when they can expect your website to come back up as well as a sign-up form that allows users to receive a notification when your website is back up.

With website monitoring, you can ensure every page on your site is up and running at all times. And if any of them go down, you’ll be the first person to know. That way, you can fix the problem before it impacts any of your customers.

The technical process of website monitoring #

Obviously, website monitoring monitors your website. But, how does that process actually work? It’s simpler than it sounds. Basically, the software uses an automated system to regularly check every page on your website and ensure an end-user could access them.

The request-response cycle

Let’s dig into the details of how that automated system works. It continuously sends your website server HTTP requests triggering the server to send an HTTP response back. That HTTP response includes a status code telling the monitoring service the current status of the page. Those HTTP status codes include:

• HTTP 2xx: A response with a HTTP status code in the 200s is the expected response. It means the page is up and running.
• HTTP 3xx: This means the page is redirecting traffic to a different page. This could be expected, but not always.
• HTTP 4xx: When there’s a problem with the code in one of your pages that prevents it from being displayed.
• HTTP 5xx: When your server cannot fulfil an HTTP request, like an issue with your website host or servers.

Receiving the status code for every page on your site is the basics of website monitoring. These services can do a lot more, though. For instance, they can also track how long it takes to fulfil a request so you can monitor the performance of your website.

That way, you can tell if a large uptick in traffic slows down your website which could mean you need to invest in faster server infrastructure.

You can even trigger an on-demand check right after a deploy.

Types of website monitoring #

The specialty tasks website monitoring can perform, like the performance testing we just mentioned, depends on the type of website monitoring provided. Here are the main types of website monitoring you should look for when signing up for a website monitoring service:

Uptime monitoring

This is the main type of website monitoring. It uses the process we discussed earlier with making an HTTP request and then analyzing the response code. However, more advanced uptime monitoring doesn’t just check that your website is up, it can also check for specific features in your website. Other services allow you to check that your website is up in specific regions or globally by making requests from localized servers.

It is not just GET, you can monitor POST, PUT and PATCH endpoints too.

SSL certificate monitoring

We’ve all gone to a site and saw the “your connection is not private” warning. That’s what happens when a site has an SSL certificate error. And just like you, that warning turns traffic away immediately. So, while receiving an HTTP response code, quality monitoring services also check the status of your SSL certificate and notify you of any errors. That way, you can fix any issues before your visitors notice them.

Broken link detection

Every website uses internal links and if you don’t, you should. But when you delete a page, it’s hard to find every single time you linked to that page on other pages. Or, if you change the URL of a page, finding every instance of a link to that URL can be a challenge. Broken link detection finds these issues for you while scouring your website. The service goes through each link on the page and makes sure it returns a valid HTTP response code as well.

Mixed content warnings

Just like an error with your SSL certificate can send your visitors a warning, mixed content on your pages does that too. That’s when your page has an SSL certificate and uses HTTPS, but content within your page uses insecure HTTP resources. This can cause your visitors to go running just like with an SSL certificate error. Monitoring this requires your website monitoring service to load all resources on the page and assess each one’s security status.

Performance monitoring

It’s easy to tell how quickly a website responds, but the information performance monitoring services can tell you a lot more than just how long it takes for a page to load from the time you enter the URL. It breaks down your page loading time into separate categories so you can identify the bottlenecks impeding your website’s performance.

Beyond up or down, we also watch speed, see performance monitoring.

Domain monitoring

There’s a lot that can go wrong with a domain. Whether it’s about to expire or someone fraudulently updated the WHOIS records, keeping tabs on your domain is just as important as keeping tabs on your website’s performance. That’s why most website monitoring services monitor your domain as well. They look for unexpected activity, warn you when your domain is about to expire, and verify all of your email-related records.

What kind of website monitoring do you need? #

Clearly there's no shortage in different types of website monitoring, but in practice you usually don’t need to pick and choose. Tools like UptimeRobot, StatusCake, and Oh Dear are considered "all-in-one" monitoring solutions. All types of monitoring are included, coupled with a robust notification system. What's also cool and totally not self-promotional, is that Oh Dear offers unlimited status pages, Lighthouse SEO monitoring and broken links and content detection as well! I know what I'd choose!

Status page templates you can copy #

When something breaks, nobody wants to stare at an empty text box. Copy the update that fits, fill in the [brackets], and post it on your status page. Each one lists the severity to pick in Oh Dear.

Keep posting while it's broken, every 30 minutes or so, even when nothing changed. Silence makes people assume the worst.

Incident: investigating

Severity: High

Investigating issues with [service]

We're looking into reports of [errors / slow pages / failed logins] on [service], starting at [time] UTC. You might not be able to [action] right now. We'll post an update within 30 minutes, sooner if we find the cause.

Incident: cause identified

Severity: High

Cause found, fix in progress

We've found the cause: [short, plain description, e.g. a failed database update]. We're rolling out a fix now. [Service] stays [unavailable / slow] until it's live. Next update within 30 minutes.

Incident: monitoring the fix

Severity: Warning

Fix deployed, monitoring

The fix is live and [service] is working again for most of you. We're keeping an eye on it to make sure it stays that way. Still seeing [problem]? Try [refreshing / logging in again] or contact [support address].

Incident: resolved

Severity: Resolved

Resolved: [service] is back to normal

[Service] has been working normally since [time] UTC. Between [start time] and [end time], [what people saw, e.g. the dashboard didn't load]. The cause was [cause]. We've [what you changed] so it doesn't happen again. Sorry for the trouble, and thanks for your patience.

Maintenance: scheduled

Severity: Scheduled maintenance

Scheduled maintenance on [date]

On [date] from [start time] to [end time] UTC, we're [upgrading our database / moving to new servers]. During that window, [service] will be [unavailable / read-only / slower than usual]. You don't need to do anything. We'll post here when we start and when we're done.

Maintenance: in progress

Severity: Scheduled maintenance

Maintenance in progress

Our scheduled maintenance has started. [Service] is [unavailable / read-only] until about [end time] UTC. We'll update this page when we're done.

Maintenance: completed

Severity: Resolved

Maintenance completed

Maintenance finished at [time] UTC and [service] is fully available again. If anything looks off, contact [support address].

Want these one click away next time? Save them as templates in Oh Dear, here's how.

Why use a status page #

A status page is a dedicated, Oh Dear hosted site where you can communicate the current status of your website or service to your users by posting status updates. It's a simple yet powerful tool for transparency and trust-building. Here’s why you should consider having one.

• Keep Users Informed: Communicate issues, such as downtime, slow performance, or scheduled maintenance, directly to your users. Instead of leaving them guessing or frustrated, they’ll know exactly what’s going on.
• Reduce Support Overhead: Proactively provide information and reduce the number of support tickets when issues arise.
• Build Trust: Transparency is key. Being open about issues builds trust with your users - a status page demonstrates that you are committed to keeping them in the loop.
• Improve User Experience: Users are more understanding of problems when they know what's happening and what to expect. By managing expectations you show that you’re on top of the situation.
• Centralised Communication: During an incident, a status page acts as a single source of truth, where all updates are published in real time. This ensures consistent messaging across all your communication channels.

Why Use Status Page Update Templates? #

Things go wrong. While there is pressure to resolve the issue, there is also pressure to communicate quickly and accurately. With our new templates, you can:

• Ensure Consistent Messaging: Keep your communications clear and consistent, reducing confusion for your users.
• Save Time During High-Pressure Situations: Apply a pre-made template, fill in the specifics, and publish your update in moments.
• Reduce Stress: No need to worry about drafting the perfect message; simply use a template and modify placeholders as needed.

Key Features #

Here’s why we think Status Page Update Templates are awesome.

• Named Templates: Create templates that are easy to find and apply.
• Flexible Fields: Not all fields are required—you can save a template with just a description or any other combination that suits your needs.
• Quick Apply: Simply select a template when creating a new update then make any necessary adjustments before publishing.
• API Support: Manage your templates with new API endpoints for listing, creating, updating and deleting templates.

Create your first template #

Getting started with Status Page Update Templates is easy:

1. Go to the 'Status Pages' section in your Oh Dear dashboard menu and select a status page to edit.
2. Select 'Status updates' in the sidebar.
3. Select 'Templates' tab.
4. Select 'Create Template'.
5. Fill in the name and any title, description, text, and severity level you need to use.
6. Click 'Save Template'.

When an incident occurs, you can select a template from the list while creating a status update, and modify any placeholders as needed.

API #

If you're looking to automate your status updates, we’ve got you covered! The Oh Dear API now supports managing status page templates with the following endpoints:

• List Templates: Fetch all saved templates.
• Create Templates: Add a new template with the desired fields.
• Update Templates: Modify existing templates.
• Delete Templates: Remove templates you no longer need.

Read the full API documentation to help you start managing your template integrations!

A More User-Friendly Approach #

We believe this new feature will make it easier for you to communicate effectively with your users. Whether you’re a small team managing one site or a large team overseeing multiple domains, having a set of templates ready can make all the difference during critical moments.

That root certificate is set to expire in a few hours. Any certificates still signed by that root will no longer be valid. But luckily, that shouldn't form a problem for most Let's Encrypt users.

For a while now, new SSL issuances by Let's Encrypt have issued certificates against DST Root CA X3 (the one that is about to expire) and ISRG Root X1. The former will expire, the latter remains valid for years to come.

If Oh Dear detects you're sending along the older, about to expire, intermediate certificate in your certificate chain, we'll notify you to take action:

The fix would be to re-issue certificates or manually change the certificate chain in your configuration that you're sending along via your webserver.

While this will only affect much older devices (ie: iPhone 4 or older), we feel it's important we notify our clients should this affect them.

For more details, we kindly refer to the official Let's Encrypt explanation of their expiring root certificate.

In this post, we'll do a technical deep-dive into how we found this problem!

Why we found it in the first place #

While we're a traditional website monitoring service in some aspects, we do a couple of things differently from our competition (and I'm not just talking about our crawler that looks for broken links & mixed content).

Our SSL monitoring is fairly intense. We fetch and validate the SSL TLS x.509 certificates every 30 minutes, for each site, to report on changes and problems.

With dozens of checks a day for the SSL certificate alone, it means we have a lot of opportunities to catch errors related to your site certificates.

This allows us to quickly identify situations where a certificate has changed (use case: did you renew all the SANs on the certificate?) or one that has incorrectly been replaced.

What happened here? #

In this case, we received reports from multiple of our clients that our certificate change reporting "seemed wrong". We were sending alerts to clients that their certificates had changed, but they could never quite verify this.

When we connect to a host to do our in-depth certificate analysis, we fetch as much information as we can. It's similar to how OpenSSL's client would do it:

$ openssl s_client \
	-connect clientsite.tld:443 \
	-servername clientsite.tld \
	2>/dev/null \
	| grep 'CN'

 0 s:/CN=clientsite.tld
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
subject=/CN=clientsite.tld
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3

This is what you'd expect: a connection to the host clientsite.tld, where we make clear that we want to retrieve the certificate information using SNI - Server Name Indication.

The above example is a functioning one, it correctly returned us the certificate of the site, and we could verify everything was intact.

However, every once in a while, this happened:

$ openssl s_client \
	-connect clientsite.tld:443 \
	-servername clientsite.tld \
	2>/dev/null \
	| grep 'CN'

 0 s:/OU=Domain Control Validated/CN=*.cdnprovider.com
   i:/C=US/ST=State/L=Location/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
 1 s:/OU=Domain Control Validated/CN=*.cdnprovider.com
   i:/C=US/ST=State/L=Location/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
 2 s:/C=US/ST=State/L=Location/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
   i:/C=US/ST=State/L=Location/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
 3 s:/C=US/ST=State/L=Location/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
   i:/C=US/ST=State/L=Location/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
ct6W1bEcW10r8Nvp0lZrYmdIn//gn4ltYpiWtRCRROEa81WO5lNDL5NO5gJ1I6CN
subject=/OU=Domain Control Validated/CN=*.cdnprovider.com

We requested the certificate for domain clientsite.tld but got back a certificate for *.cdnprovider.com.

Clearly this shouldn't happen!

As a result, our alerts got sent to the client, because their site was being served by an incorrect certificate.

Changes made at Oh Dear to help verify this #

When we received the first reports, it wasn't easy to troubleshoot this. You see, we connect to the hostname clientsite.tld, but that might be served by several IP addresses using DNS Round Robin.

$  dig +short +noshort clientsite.tld
clientsite.tld.		66	IN	A	1.2.3.4
clientsite.tld.		66	IN	A	5.6.7.8

In fact, that's how most CDNs operate: you point your domain to a CNAME, and they present multiple anycasted IPs, each capable of serving your site.

Our problem was that we didn't present the IP that we used for this certificate check back to you, the user. So how could you know which IP served the wrong certificate?

As part of our debugging, we made this available to all users in the detailed view of their certificate report:

To do this, we contributed to the upstream SSL package to expose the remote address IP, since this wasn't available out of the box.

/* ... */
$response['remoteAddress'] = stream_socket_get_name($client, true);

But, this wasn't enough to easily pinpoint the issue.

Working together with the CDN #

You see, many of the IPs of a CDN are "anycast" IPs: you connect to an IP address, but there might be multiple servers worldwide responding to that very same IP.

The short version is: the one closest to you (in BGP-terms) will provide you the answer. But for the CDN provider, there's no way to uniquely identify which of their thousands of servers actually responded.

We worked together with them to provide all technical details of our checks and sufficient example/test cases to, at first, convince them of this problem. Then, our examples helped them to troubleshoot and resolve the problem on their end.

I wanted to let you know that we found two edge servers that were mis-behaving and causing the issue you were seeing. We did a full sweep of the thousands of servers we manage and confirmed no others were in this bad state.

-- Director of Customer Support

We're thrilled to see the widespread implications of our paranoid certificate monitoring, and that, as a result, we helped thousands of sites worldwide. Even those not using Oh Dear!

Now, the morning after, we're well rested and in good shape to do a proper write-up on the matter.

Getting a list of all domains to check #

As part of our uptime monitoring, users can add a site to Oh Dear with specific URL parameters. So in order to get a list of domains we needed to verify, it wasn't as simple as:

SELECT domain FROM sites;

Instead, we used Laravel's lazy collections to quickly filter all teams with active subscriptions and extract the relevant domain name.

Team::cursor()
    ->filter(fn (Team $team) => $team->hasActiveSubscriptionOrIsOnGenericTrial())
    ->flatMap(fn (Team $team) => $team->sites)
    ->map(fn (Site $site) => $site->domain())
    ->each(fn (string $output) => echo $output . PHP_EOL);

This produced a new-line separated list of domains that we need to check.

Let's save those in domains.txt, since we're moving to some CLI tricks now.

Retrieving the serial for each certificate #

Now we find the active Serial Number for each of those certificates. It involves connecting to each site over SSL/TLS, getting the certificate and saving the Serial Number.

The original idea came from a Hacker News comment, we modified it to get some better error handling and control of the output.

# Create a directory to hold all serial numbers
mkdir -p serials

# Loop all domains, connect and fetch the serial
for i in $(cat domains.txt); do
  echo "Connecting to $i ... "

  (
    openssl s_client -connect $i:443 -servername $i -showcerts < /dev/null 2> /dev/null |
    openssl x509 -text -noout |
    grep -A 1 "Serial Number" |
    tr -d : |
    tail -n 1
  ) | tee serials/$i;

done

The openssl s_client connects to the domain (using Server Name Indication (SNI) with the -servername option) and lists all certificates.

Now, in serials/*, we have a directory full of domain names and their corresponding certificate serial.

Combining all serials #

We'll make a single list with all the serials we need to check. This way, we can optimize our grep commands for later.

$ cat serials/* | tr -d " " | sort | uniq > serial-numbers.txt

The file serial-numbers.txt is now a gigantic list of serial numbers.

Finding the serials in the 1.2GB text file #

Let's Encrypt has released a text-file with all affected certificates. This file includes the Serial Number (which we now have) together with all domains/SANs on the certificate.

Our first attempt was to simply grep our way through the file for each serial found. But grep is single-threaded, so we could only utilize a single CPU core for searching through a pretty big file.

This was taking too long, so we quickly adapted our method and started to search through the log in parallel.

Lucky for us, we started preparing a new set of servers for our crawlers that check for broken links last week. Those servers were still idling as they aren't in production yet. This was the perfect time to use that spare capacity.

First, we split the big file of serials (called serial-numbers.txt) in many equal pieces.

$ split -l 1000 serial-numbers.txt

This gives us a list of many files, all with 1000 serial numbers in it. The file naming is predictable:

$ ls -l x*
-rw-rw-r-- 1 immutable immutable 2672 Mar  3 20:27 xaa
-rw-rw-r-- 1 immutable immutable 2948 Mar  3 20:27 xab
-rw-rw-r-- 1 immutable immutable 2948 Mar  3 20:27 xac
-rw-rw-r-- 1 immutable immutable 2960 Mar  3 20:27 xad

In order to utilize all our cores, we used each file as the pattern input to grep and sent the job to the background for processing.

for file in $(ls x*); do
	\grep -P "$(cat $file | tr "\n" "|" | sed -e 's/|/\|/g' | sed -e 's/|$//' )" ../ssl-cert/caa-rechecking-incident-affected-serials.txt >> results.txt &
done

That rather ugly-looking tr & sed pipeline in there transforms the input file from a new-line separated list of serials, to a |-separated list. This is used in grep to indicate the "or" statement, any line may match.

In the long form, it turns our input of this:

$ cat xaa
01009ba...
0111839...
011539e...
0135d43...

... into this:

01009ba...|0111839...|011539e...|0135d43...

Because we sent each grep command to the background in our for-loop, using the & at the end of the command, we now have many grep's running in parallel.

What followed was, to me as a sysadmin, a thing of beauty. ?

The crunching continued for a while, and we now had a list of affected serials stored in results.txt.

At this point, things were getting a bit late, so we resorted to even weirder Bashness to match these serials back to the domain names.

Matching the serials back to domains #

We loop each affected serial and match it back to the domain:

for line in $(cat results.txt ); do
	\grep $line serials/*;
done |
	awk '{print $1}' |
	sed 's/\// /' |
	awk '{print $2}' |
	sed 's/:/ /' |
	awk '{print $1}'

Looking at it now, awake, it could've been much cleaner. But, it got the job done! We now have a list of domain names of clients we need to notify.

Sending the mails to clients #

To inform our clients, we resorted back to PHP. This allows us to send the notification e-mails in our own style/branding.

Mail::to($users)->send(new LetsencryptRevokedMail($domain));

It uses the power of Laravel Mailables to make this really easy.

A rush job because time was against us #

We didn't do as a clean a job as we'd normally do. There were no tests, no clean integrations, and most of it was hacked together on very short-notice.

But, there wasn't much choice. The list of affected certificates was released yesterday, and within 48 hours the revocation was to take place. It was up to us to notify our users asap. After all, those affected still needed to renew their certificates!

We're happy to see the list of affected domains beforehand though. The normal procedure is that the revoced certificates end up in Certificate Revocation Lists (CRL), but at that point the revocation has already happened.

This allowed us to be proactive and inform clients ahead of time!

Our interview with Marbles #

Hi Rias, who are you and what does your business do?

We are Marbles, an Antwerp based full service agency that does everything from creating identities to marketing & communications, business development and coaching. Of course creating websites and web applications is part of our services and to make sure they're the best they can be we use Oh Dear! for the monitoring.

Cool! Why did you chose Oh Dear! as your monitoring solution?

It includes all the checks we need for a very fair price.

Who did you have before?

We used Pingdom, but only for the basic uptime checks.

What made you switch?

The detection of mixed content & broken links and the https certificate expiration checks on top of the uptime monitoring.

How does Oh Dear! help you provide better service for your clients?

By alerting us of issues before anyone could possibly notice it manually. This allows us to fix issues before our clients are even aware of it, or if they are we can tell them we're already working on a fix.

What's your favorite feature of Oh Dear?

All the extra checks on top of basic uptime monitoring with Slack notifications and webhooks to receive reports.

Is there a killer feature we're missing that could help you?

Setting up notification "groups". We have clients who have an SLA with us, they need very quick solutions and we use the SMS notifications for them, for other clients we only set up the Slack Notifications.

It's bothersome to set up the SMS notifications individually for each site that needs them, and if another person needs to receive these, or the phone numbers need to get changed, we need to do this for each site manually.

In response: this is indeed a very good idea still missing, we're taking this up in our development to see if we can introduce an elegant solution to this problem. Thanks for letting us know!

What makes Oh Dear! special to you? #

We'll be reaching out to more of our customers over the next months to find out why they chose us over the (many) competitors out there. We'd love to know what pursuaded them and what items we still need to address to serve our customers even better.

If you want to share your thoughts, feel free to reach out to us on Twitter at @OhDearApp.