AI monitoring

AI monitoring lets you verify virtually anything about your websites and services that traditional monitoring might miss. Instead of pre-defined checks, you describe what you want to verify in plain English.

⚠️ This check is currently in closed beta. If you'd like to try it out, please apply to get access to the AI monitoring beta.

How AI monitoring works #

Unlike traditional website monitoring that checks for specific metrics like uptime or performance, AI monitoring works by describing your check in plain English. Here's an example:

On example.com/contact fill in the contact form with dummy values. If the submission seems unsuccessful, mark this check as failed and send me a notification. If it fails, also take a screenshot and show it in the check results.

The AI has access to the web. It can click and type on webpages, take screenshots, and can also take a look at the message on the developer console of a page. Additionally, it can also look up DNS records.

Getting started #

To use AI monitoring, create a new monitor in Oh Dear and select "AI" as the monitor type. Enter the URL you'd like to monitor, and we'll analyze the page and suggest a prompt for you. You can tweak the suggestion or write your own from scratch. You can also choose how often the check runs (from every minute to once every two months).

For the first 5 AI calls, you can make use of Oh Dear's internal AI account for free. After that, you'll need to set up your own API key from a supported AI provider in the AI configuration screen. We currently support:

• OpenAI
• Anthropic
• Gemini
• xAI (Grok)

Auto-suggested prompts #

When you add a new AI monitor, we'll analyze the page you want to monitor and suggest a prompt tailored to your site. We look at the structure of the page - navigation, hero sections, product listings, forms, and more - and generate a prompt that covers the most important things to check.

You're always free to edit the suggestion, replace it entirely, or start from scratch. Think of it as a starting point that saves you from staring at a blank text box.

How to construct a good prompt #

When writing prompts, be as specific as possible about what you want to verify. The more detail you provide, the better the AI can understand your requirements and provide accurate results. Your prompt should have these elements:

• The URL to check
• An exact description of what to look for
• What to do if the conditions are met or not met (e.g., send a notification)
• A clear description of what should be included in the results

Here is an example of a good prompt:

Check that our homepage https://example.com displays exactly 5 products, each with a title, image, and price. Ensure there is a call-to-action button labeled "Shop Now" that links to the /shop page.

If all these conditions are met, the check is ok, and you don't need to send me a notification. If any condition is not met, the check fails, and you need to send me a notification explaining what is wrong. Also take a screenshot of the page.

In general, the more specific you are, the better the results will be.

More prompt examples #

Here's some inspiration for other prompts you can use. The AI can check live websites in real-time. It is also able to use our notification system.

On example.com/contact fill in the contact form with dummy values. If the submission seems unsuccessful, mark this check as fail and send me a notification.

If there are any warnings or errors on the developer console of mysite.com, fail the check and send me a notification. You can ignore any messages from CloudFlare.

Make sure the homepage of mysite.com lists at least five products, includes a login link, and has a registration page link. Send me a notification if any of these elements are missing.

The AI has access to DNS records, so you can also use it to verify your DNS configuration:

Verify that all MX records for spatie.be point to Google. If they don't, send me a notification and show all current MX records in the result.

We support POST requests with custom payloads and headers:

Send a POST request to mysite.com/api/my-endpoint with this payload: first_name: "John" last_name: "Doe" Include this HTTP header: my-custom-header: my-custom-header-value The response code should be 200. Send me a notification if the response code is anything other than 200.

Of course, you can also use it to verify your robots.txt or sitemap.xml files:

Verify my robots.txt on https://example.com allows Google but blocks ChatGPT's AI crawler.

Check the spelling on https://example.com. If there are any spelling mistakes, send me a notification.

Monitor the RSS feed on https://example.com/blog. If there is a new post, send me a notification.

AI tools and capabilities #

The AI monitoring system has access to several tools to help verify your requirements:

• Real browser interaction: The AI can click, type, fill in forms, select options, and navigate across pages - just like a real user. A full browser session is started, so multi-step flows (like adding a product to a cart or submitting a form) work as expected. JavaScript is fully executed, and the AI can also inspect console messages and network requests made during the session.
• Web page fetching: Can retrieve and analyze any web page content. You can specify which HTTP verb to use (GET, POST, etc.) and add custom headers to the request.
• Screenshot: The AI can take a screenshot during a browser session and include it in the check results, so you can visually verify what it saw.
• DNS record lookup: Can query and verify DNS configurations like MX, A, CNAME, and TXT records.
• Date/time awareness: The AI knows the current date and time, which is useful for checking content freshness (e.g. "the latest blog post should be less than 7 days old").
• Previous check results: Each run, the AI has access to its previous response. This helps it give consistent results across runs and detect changes over time.

Limitations and considerations #

• The AI can fetch up to 5 web pages per check execution using the simple fetch tool. For browser interactions, a maximum of 50 browser actions can be performed. Do not ask it to crawl an entire website, as your check will be halted.
• Each check execution has a maximum time limit of 5 minutes
• AI monitoring is designed for functional and content verification, not performance monitoring, for which our traditional checks are better suited
• Prompts should be specific enough for consistent results across multiple runs

Check results #

After each run, the AI provides a detailed response explaining what it checked and what it found. This response is displayed on the check results page as formatted markdown, and can include screenshots.

Each check produces one of these results:

• OK: Everything matches your expectations
• Failed: Something doesn't match - the AI will explain exactly what went wrong
• Error: The check itself couldn't run (e.g. an API timeout or provider issue). This is different from a failed check - it means the AI didn't get a chance to verify your site at all