Application health monitoring

Using Oh Dear you can monitor various aspects or your application and server. This way you could get alerts when:

• disk space is running low
• the database is down
• Redis cannot be reached
• mails cannot be sent
• there are many application errors in a small timeframe (via Flare)
• a reboot of your app is required
• ...

You can monitor any aspect of your app that you want.

Getting started

Responding with health check results

Oh Dear will not run any code inside your application or server. Instead, you should perform the checks yourself. Oh Dear will send an HTTP request to your application to a specific endpoint. Your application should respond with JSON containing the result of health checks.

The request that Oh Dear sends to your app will contain a secret value in the oh-dear-health-check-secret header. To make sure that a request really comes from Oh Dear, you should verify if the secret value is correct.

Both the endpoint to which the request will be sent and the secret in the header can be specified the application health check settings.

Integration in your application

Here's how you can let your application respond with health check results in the expected format.

• Laravel
• Any PHP application
• All other languages

Viewing application health results in Oh Dear

Oh Dear will check the health results endpoint of your app every few minutes. You can view the results on the application health check screen.

You can click any of the checks to see the history of results for that particular check.

We only store the history for a couple of days. If you need to keep the history longer, consider using our API to fetch the results, and save them on storage of your own.

Getting notified of health check issues

Whenever a health check problem is detected (or if fixed) Oh Dear can notify you. To do this, you must enable the application health notifications at the site or team level.

For every check that fails, we'll send you a notification. To avoid swamping you in notifications, we'll only send one notification per failing health check per hour. We'll also send you a notification when a problem has been fixed.

Adding and removing health checks

Any health check result that your server responds with, will be added to Oh Dear. To remove a specific health check from Oh Dear, simply let your server not send the result to us anymore, and we'll remove that check from our system too.

Laravel

You can start monitoring the health of a Laravel application by using the spatie/laravel-health package.

Using this package you can monitor the health of your application by registering one of the available checks. Out of the box it can monitor disk space, if Horizon is running, if your application is sending a lot of application errors (through Flare), and much more.

Here's an example where we'll monitor used disk space.

// typically, in a service provider

use Spatie\Health\Facades\Health;
use Spatie\Health\Checks\Checks\UsedDiskSpaceCheck;

Health::checks([
    UsedDiskSpaceCheck::new()
        ->warnWhenUsedSpaceIsAbovePercentage(70)
        ->failWhenUsedSpaceIsAbovePercentage(90);
]);

Installation and registering checks

In the docs of spatie/laravel-health, you'll find instructions on how to install the package and register your first check. We highly recommend reading the full docs of the package, so you know what it's capable off.

Adding a health check endpoint to your Laravel app

Oh Dear will send an HTTP request to your application to a specific endpoint to get health check. Your application should respond with JSON containing the result of health checks.

You can add such an endpoint using the spatie/laravel-health package. To do this, must configure the ohdear_endpoint_key in the health config file.

You can publish that health with this command:

php artisan vendor:publish --tag="health-config"

These are some of the default values in the published health config file.

// in app/config/health.php

/*
 * You can let Oh Dear monitor the results of all health checks. This way, you'll
 * get notified of any problems even if your application goes totally down. Via
 * Oh Dear, you can also have access to more advanced notification options.
 */
'oh_dear_endpoint' => [
    'enabled' => false,

    /*
     * When this option is enabled, the checks will run before sending a response.
     * Otherwise, we'll send the results from the last time the checks have run.
     */
    'always_send_fresh_results' => true,

    /*
     * The secret that is displayed at the Application Health settings at Oh Dear.
     */
    'secret' => env('OH_DEAR_HEALTH_CHECK_SECRET'),

    /*
     * The URL that should be configured in the Application health settings at Oh Dear.
     */
    'url' => '/oh-dear-health-check-results',
],

To get started:

• set the enabled config option to true
• add a secret (we recommend putting it in the .env file, just like you would do for any application secret or password)
• optionally customize the url where the health check endpoint will be registered.

Adding health checks

Now that you have registered an endpoint, let's add some checks to your app. To learn how to do this, head over to the Registering your first check page in the laravel-health docs.

Configuring the health check at Oh Dear

In the application health check settings screen at Oh Dear, you should fill in the URL and secret that you specified in the health config file.

Avoid stored results and sending notifications locally

Oh Dear will notify you when something goes wrong with one of your health checks, so you can safely disable the notifications sent by the package itself.

You can set the notifications.enabled option to `false.

'notifications' => [
    /*
     * Notifications will only get sent if this option is set to `true`.
     */
    'enabled' => true,

If you don't want to keep health results in your local application, set the result_stores option in the health config file to the InMemoryHealthResultStore.

// in app/config/health.php

'result_stores' => [
     Spatie\Health\ResultStores\InMemoryHealthResultStore::class,
],

Any PHP application

Oh Dear can monitor the health of any PHP, no matter which framework it uses or language is written in.

We will not run any health check from our end. Your application or infrastructure should run any desired checks. The results of the checks should be made available at an HTTP endpoint.

Every couple of minutes Oh Dear will visit that endpoint to get the check results. The endpoint URL can be configured at the application health settings screen.

The request that Oh Dear sends to your app will contain a secret value in the oh-dear-health-check-secret header. To make sure that a request really comes from Oh Dear, you should verify if the secret value is correct.

Building the health checks results JSON in PHP

That endpoint must return the health check results as JSON in a specific format. That format is described on this page.

To help you construct the required JSON, you can use the ohdearapp/health-check-results package.

Here's an example that shows how you can create the JSON that Oh Dear expects.

$checkResults = new CheckResults(DateTime::createFromFormat('Y-m-d H:i:s', '2021-01-01 00:00:00'));

$checkResult = new CheckResult(
    name: 'UsedDiskSpace',
    label: 'Used disk space',
    notificationMessage: 'Your disk is almost full (91%)',
    shortSummary: '91%',
    status: CheckResult::STATUS_FAILED,
    meta: ['used_disk_space_percentage' => 91]
);

$checkResults->addCheckResult($checkResult);

echo $checkResults->toJson();

This will output this JSON:

{
    "finishedAt": 1609459200,
    "checkResults": [
        {
            "name": "UsedDiskSpace",
            "label": "Used disk space",
            "notificationMessage": "Your disk is almost full (91%)",
            "shortSummary": "91%",
            "status": "failed",
            "meta": {
                "used_disk_space_percentage": 91
            }
        }
    ]
}

All other languages

Oh Dear can monitor the health of any application, no matter which framework it uses or language is written in.

We will not run any health check from our end. Your application or infrastructure should run any desired checks. The results of the checks should be made available at an HTTP endpoint.

Every couple of minutes Oh Dear will visit that endpoint to get the check results. The endpoint URL can be configured at the application health settings screen.

The request that Oh Dear sends to your app will contain a secret value in the oh-dear-health-check-secret header. To make sure that a request really comes from Oh Dear, you should verify if the secret value is correct.

Health check results format

That endpoint must return the health check results as JSON in a specific format.

The JSON should have two toplevel properties:

• finishedAt: a unix timestamp indicating when these check did run on your server.
• checkResults: an array with check results. This array can have 50 items maximum.

A check result should have these properties, they are all required:

• name: this should be a unique value to identify this check. We'll use this value to synchronise the check results in our database.
• label: a string describing your health check that we will display at the application health checks list.
• status: a string that indicates what the result of the health check is. Check below for possible values.
• notificationMessage: when a check results in warning or error, we'll use this string in the notification that we'll send to you. we'll also display it at the application health checks list.
• shortSummary: a very short summary of the result of your check that will be displayed at the application health checks list.
• meta: an array with keys and values that contain extra information about the check. You can send a maximum of 20 items in this array.

These are the possible values for status:

• ok: the check ran ok
• warning: the check is closed to failing
• failed: the check did fail
• crashed: something went wrong running the check itself
• skipped: the check wasn't performed at all

Here's an example response that your server should return:

{
    "finishedAt": "1638879833",
    "checkResults": [
        {
            "name": "UsedDiskSpace",
            "label": "Used Disk Space",
            "status": "failed",
            "notificationMessage": "The disk is almost full (91% used)",
            "shortSummary": "91%",
            "meta": {
                "disk_space_used_percentage": 91
            }
        },
        {
            "name": "Redis",
            "label": "redis",
            "status": "ok",
            "notificationMessage": "",
            "shortSummary": "Available",
            "meta": []
        }
    ]
}