There There
Twelve checks on one site, before the first customer.
While There There was still in closed beta, the team pointed one Oh Dear monitor at there-there.app and turned on every check. Alerts go to #coders, the Slack channel the developers already read.
- Bitmovin
- HBO Nordic
- Obsidian
- Laravel
- Fathom Analytics
- PHP 8
- Stanford University
- Takeaway.com
- IGN
- VRT NWS
- spatie
June 9, in numbers
Fixed before support hours started.
2 min from the first failed check to the Slack alert
43 min for the whole incident, start to recovery
0 customers who noticed
0 emails lost. They queued and were redelivered
Before launch
One monitor. Every check on from day one.
There There is an AI-powered helpdesk built by Spatie, the Belgian team behind many of the open-source packages Laravel developers install. If a helpdesk goes down, its customers' support goes down with it.
oh dear
there-there.app
12 checks
Uptime
Cron jobs
Certificates
Application health
Broken links
Sitemap
Mixed content
DNS
Lighthouse
Domain
Ports
DNS blocklist
Notifications: Slack #coders
June 9, 08:54
The server looked fine. Nobody outside could reach it.
Nightly automatic security updates upgraded systemd and restarted the server's networking. The server never crashed. It still received traffic, but none of its replies made it back out.
there-there.app, 08:54 to 09:37 Same server, same minute
Seen from the server
All healthy
OK
Load
Normal
OK
Processes
Running
OK
Application
Healthy
OK
Server uptime
75 days
oh dear
Seen by Oh Dear
there-there.app
Failed
GET /
Timed out after 5s
Failed
Bytes returned
0
Failed
Replies leaving the server
None
Failed
Status
Unreachable
A 5-second timeout with zero bytes back is what it looks like when replies never leave the machine. That line in Oh Dear's check log was the tell. With their hosting provider UpCloud, the team traced the cause, rebooted to reset the network, and there-there.app came straight back. More on how the outside check works: website uptime monitoring.
Two minutes in
The failed deploy pointed them at GitHub.
The outage started in the middle of a deploy, so the first error the team saw was the wrong one.
The deploy failed first.
Freek Van der Herten was deploying when the script failed on the "clone GitHub repo" step. His first guess was a GitHub problem. Then Oh Dear posted in #coders that there-there.app was down, and the question changed from "what's wrong with GitHub?" to "what's wrong with our server?"
Deploy script
Clone GitHub repo failed
08:54
Oh Dear to #coders
there-there.app is down
~08:56
Oh Dear noticed the server was down and told us in Slack within a few minutes, before any customer noticed.
Freek Van der Herten, Founder, Spatie
Freek is also a co-founder of Oh Dear.
What customers saw
No customer reported it.
It happened early in the morning and was over before support hours got going. Mail sent to There There queued upstream and was delivered in full once the server came back. Nothing was lost.
The incident is on There There's public status page, hosted by Oh Dear: detected, investigated, resolved. Developers buying a helpdesk trust a logged incident more than a perfect record.
oh dear
Status page
60 days ago
99.951% uptime
Today
One incident, on the record
No feature tiers
Every check.
Every plan.
The uptime check that caught this outage is on every plan, next to the other eleven There There runs. Choose a plan by how many sites you monitor, not by which checks you need.
Start with everything on.
Turn on every check from day one, the way There There did.
- Uptime checked from outside your server
- Every monitoring feature included
- Slack alerts and public status pages
- No credit card required
10-day free trial. No credit card.
Questions about the outage.
What broke, why only an outside check saw it, and what it cost There There.
What broke on There There's server?
Nightly automatic security updates upgraded systemd and restarted the server's networking. The server kept receiving traffic, but its replies never got out. A reboot reset the network.
Why didn't anything on the server notice?
From the inside, load, processes and the application all looked normal, and the server had been up for 75 days. The fault was in outbound traffic, so it only showed up for something checking the site from outside the server.
How fast did Oh Dear alert the team?
The first check failed at 08:54. Oh Dear posted the site as down in Slack around 08:56. The site was back at 09:37, 43 minutes after the first failure.
Was any data or mail lost?
No. Mail sent to There There queued upstream and was delivered in full after the server recovered. No customer reported the outage.
Which Oh Dear checks does There There use?
All twelve: uptime, certificates, broken links, mixed content, Lighthouse, ports, cron jobs, application health, sitemap, DNS, domain and DNS blocklist. Alerts go to the #coders Slack channel.
