Content Negotiation for AI Agents: Serving Markdown

Our application now supports content negotiation to better serve AI agents and LLMs. We've added the ability to return Markdown instead of HTML when requests include the Accept: text/markdown header.

This enhancement allows AI agents to directly consume the raw Markdown content of our posts, simplifying parsing and improving efficiency.

The Problem

Previously, our application served HTML content by default. While suitable for browsers, HTML can be verbose and require significant parsing for AI agents that primarily need the textual content. Serving Markdown directly reduces overhead and improves the agent's ability to quickly extract relevant information.

The Solution

We've implemented content negotiation based on the Accept header. When a request includes Accept: text/markdown, the application now returns the content in Markdown format. This leverages the fact that our posts are already stored as Markdown in the database, eliminating the need for on-the-fly conversion.

Here's a simplified example of how this might be implemented using middleware in a Laravel application:

<?php

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\Request;
use Symfony\Component\HttpFoundation\Response;

class ContentNegotiationMiddleware
{
    public function handle(Request $request, Closure $next): Response
    {
        if ($request->header('Accept') === 'text/markdown') {
            return response($this->getMarkdownContent(), 200, ['Content-Type' => 'text/markdown']);
        }

        return $next($request);
    }

    private function getMarkdownContent(): string
    {
        // Logic to fetch Markdown content from the database
        // For example:
        // $post = Post::find(1);
        // return $post->content;

        return '# Example Markdown Content\n\nThis is an example of Markdown content served to AI agents.';
    }
}

In this example:

• The ContentNegotiationMiddleware checks for the Accept: text/markdown header.
• If the header is present, it fetches the Markdown content and returns it with the appropriate Content-Type.
• Otherwise, it passes the request to the next middleware in the stack.

To register the middleware in app/Http/Kernel.php:

    protected $middlewareGroups = [
        'web' => [
            // Other middleware
            \App\Http\Middleware\ContentNegotiationMiddleware::class,
        ],
    ];

Discoverability

To further enhance discoverability for AI agents, we've also added a llms.txt file (similar to robots.txt) that provides information about the available content formats. We ensure that robots.txt also points to llms.txt.

Conclusion

By implementing content negotiation and providing Markdown content directly, we've optimized our application for AI agent consumption. This improves efficiency, reduces parsing overhead, and enhances the overall experience for AI-powered tools interacting with our content.