AI monitoring

AI monitoring in Oh Dear allows you to use artificial intelligence to verify virtually anything about your websites and services that traditional monitoring might miss. Instead of pre-defined checks, you provide natural language prompts describing what you want to verify.

⚠️ This check is currently in closed beta. If you'd like to try it out, please apply to get access to the AI monitoring beta.

How AI monitoring works #

Unlike traditional website monitoring that checks for specific metrics like uptime or performance, AI monitoring works by describing your check in plain English. Here's an example "On example.com/contact fill in the contact form with dummy values. If the submission seems unsuccessful, mark this check as fail and send me a notification. If it fails also take a screenshot and show it in the check results."

The AI has access to the web. It can click and type on webpages, take screenshots, and can also take a look at the message on the developer console of a page. Additionally, it can also look up DNS records.

Getting started #

To use AI monitoring, you need to create a new monitor in Oh Dear and select "AI" as the monitor type. You then provide a prompt that describes what you want to verify. You can choose how often the check runs (from every minute to once every two months).

For the first 10 AI calls, you can make use of Oh Dear's internal AI account for free. After that, you'll need to set up your own API key from a supported AI provider in the AI configuration screen. We currently support:

• OpenAI
• Anthropic
• Gemini

How to construct a good prompt #

When writing prompts, be as specific as possible about what you want to verify. The more detail you provide, the better the AI can understand your requirements and provide accurate results. Your prompt should have these elements:

• The URL to check
• An exact description of what to look for
• What to do if the conditions are met or not met (e.g., send a notification)"
• A clear description of what should be included in the results

Here is an example of a good prompt:

Check that our homepage https://example.com displays exactly 5 products, each with a title, image, and price. Ensure there is a call-to-action button labeled "Shop Now" that links to the /shop page.

If all these conditions are met, the check is ok, and you don't need to send me a notification. If any condition is not met, the check fails, and you need to send me a notification explaining what is wrong. Also take a screenshot of the page.

In general, the more specific you are, the better the results will be.

More prompt examples #

Here's some inspiration for other prompts you can use. The AI can check live websites in real-time. It is also able to use our notification system.

On example.com/contact fill in the contact form with dummy values. If the submission seems unsuccessful, mark this check as fail and send me a notification.

If there are any warnings or errors on the developer console of mysite.com, fail the check and send me a notification. You can ignore any messages from CloudFlare.

Make sure the homepage of mysite.com lists at least five products, includes a login link, and has a registration page link. Send me a notification if any of these elements are missing.

The AI has access to DNS records, so you can also use it to verify your DNS configuration:

Verify that all MX records for spatie.be point to Google. If they don't, send me a notification and show all current MX records in the result.

We support POST requests with custom payloads and headers:

Send a POST request to mysite.com/api/my-endpoint with this payload: first_name: "John" last_name: "Doe" Include this HTTP header: my-custom-header: my-custom-header-value The response code should be 200. Send me a notification if the response code is anything other than 200.

Of course, you can also use it to verify your robots.txt or sitemap.xml files:

Verify my robots.txt on https://example.com allows Google but blocks ChatGPT's AI crawler$

Check the spelling on https:/example.com. If there are any spelling mistakes, send me a notification.

Monitor the RSS feed on https://example.com/blog. If there is a new post, send me a notification.

AI tools and capabilities #

The AI monitoring system has access to several tools to help verify your requirements:

• Real Browser Interaction: You can ask the AI to click or type on any web page. A session will be started, so you can use multi-step verifications. JavaScript will be executed on the page. The AI also has access to the messages on the developer console.
• Simple Web page fetching: Can retrieve and analyze any web page content. You can specify which HTTP verb to use (GET, POST, etc.). You can also instruct the AI to add extra headers to the request.
• Date/time awareness: The AI knows the current date and time for time-sensitive checks
• DNS record lookup: Can verify DNS configurations and changes
• Previous check results: The AI can use the results of previous checks to verify the current state of the website
• Screenshot: The AI can take a screenshot of the current page

Limitations and considerations #

• The AI can only fetch a couple of web pages per check execution. Do not ask it to crawl an entire website, as this you check will be halted.
• Each check execution has a maximum time limit of 3 minutes
• AI monitoring is designed for functional and content verification, not performance monitoring, for which our traditional checks are better suited
• Prompts should be specific enough for consistent results across multiple runs

Check results #

AI monitoring provides two possible results:

• OK: Everything is working as expected according to your prompt
• Failed: Significant problems found that require immediate action