Making our docs AI-friendly: a tale of two caches

Our documentation, FAQ, and blog posts can now be served as clean markdown to AI agents. Send Accept: text/markdown or append .md to the URL, and you get structured content instead of a full HTML page.

It worked great in development. Then we deployed, and two separate caching layers broke everything. Here's the journey.

The implementation (surprisingly easy) #

Just last week, we added .md suffixes to all our docs URLs and listed them in our /llms.txt. That felt like a solid approach. A week later, the convention has already moved on. Accept: text/markdown is now the preferred way. It's cleaner HTTP and works with standard content negotiation. We're keeping the .md suffixes for tools that can't set custom headers.

The actual implementation is a small trait:

trait DetectsMarkdownRequest
{
    protected function shouldReturnMarkdown(): bool
    {
        if (str_ends_with(request()->path(), '.md')) {
            return true;
        }

        if (str_contains(request()->header('Accept', ''), 'text/markdown')) {
            return true;
        }

        return false;
    }
}

Each controller just checks this early and returns markdown when needed:

public function __invoke(string $slug = ''): View|RedirectResponse|Response
{
    if (str_ends_with($slug, '.md')) {
        $slug = Str::chopEnd($slug, '.md');
    }

    if ($this->shouldReturnMarkdown()) {
        return $this->markdownResponse($slug);
    }

    // ... regular HTML response
}

That's it. Add the trait, add the early return, done. The markdownResponse() method returns the raw content with the right headers:

return response($markdown, 200, [
    'Content-Type' => 'text/markdown; charset=UTF-8',
    'X-Robots-Tag' => 'noindex',
]);

The savings #

Here's what our pages look like as HTML versus markdown:

What that means in API costs, per request for the Monitor API docs page:

Per request, the difference is small. But agents crawling multiple docs pages per session add up, and there's no reason to ship navigation chrome and script tags to something that just wants the content.

And then production happened.

Cache layer 1: Cloudflare #

Cloudflare caches by URL. It doesn't vary by Accept header. So the first browser visit cached the HTML, and every subsequent request, including Accept: text/markdown, got that cached HTML back.

First instinct: add Vary: Accept. Nope. Cloudflare Free/Pro doesn't respect Vary: Accept for non-image content.

There's a clever workaround using Transform Rules to append a query parameter to markdown requests, giving them a different cache key. Mike Olson wrote a great post about this approach for the Cloudflare Free plan.

We went simpler: a Cloudflare Cache Rule that bypasses cache entirely for markdown requests.

• Rule name: "Do not cache text/markdown requests"
• Expression: any(http.request.headers["accept"][*] contains "text/markdown")
• Action: Bypass cache

Available on all plans, including Free. Why not the Transform Rule? Markdown requests are a tiny fraction of traffic. Bypassing the CDN for those has zero noticeable impact, and the rule is dead simple. Sometimes the boring solution is the right one.

Cache layer 2: spatie/laravel-responsecache #

Fixed Cloudflare. Tested again. Still HTML. What.

We use spatie/laravel-responsecache on our docs routes. First attempt: override shouldCacheRequest() to return false for markdown. Still HTML.

Here's why. The middleware does this on every request:

if enabled() && hasBeenCached($request)
    → return cached response          // our check never runs

if enabled() && shouldCacheRequest($request)
    → store the fresh response         // only controls STORING

shouldCacheRequest() only controls storing, not serving. The cached HTML was already returned before our override ever ran.

The fix: vary the cache key using useCacheNameSuffix(), the documented way to differentiate requests:

public function useCacheNameSuffix(Request $request): string
{
    $suffix = parent::useCacheNameSuffix($request);

    if (str_contains($request->header('Accept', ''), 'text/markdown')) {
        return $suffix.'-markdown';
    }

    return $suffix;
}

HTML and markdown now get separate cache entries. Both cached, both served correctly.

We also added a Vary: Accept middleware on the docs routes. Cloudflare ignores it on our plan, but it's correct HTTP semantics for browsers and other proxies.

What we learned #

1. Vary: Accept doesn't work on Cloudflare Free/Pro. Use Cache Rules with any(http.request.headers["accept"][*] contains "text/markdown") instead.
2. shouldCacheRequest() in spatie/laravel-responsecache only controls storing, not serving. Use useCacheNameSuffix() to vary the cache key.
3. Test with all cache layers enabled. Our local dev had response cache disabled, masking the bug entirely. Worked in dev, failed in prod. Classic.
4. Caching bugs are sneaky. Valid response, wrong variant. No errors, no 500s, nothing in the logs.

Try it out #

# Regular HTML
curl -I https://ohdear.app/docs/api/monitors

# Markdown for AI agents
curl -H 'Accept: text/markdown' https://ohdear.app/docs/api/monitors

# Also works with .md suffix
curl https://ohdear.app/docs/api/monitors.md

Our llms.txt lists all available documentation. This works for feature docs, API docs, FAQ items, and blog posts. Everything a coding agent needs when integrating with Oh Dear.

If you're running Cloudflare + Laravel response cache and want to serve different content based on headers, hopefully this saves you the debugging time we spent on it.