Improving a website's visibility to search engines often involves a combination of structured data implementation and proper handling of robots.txt. This post details how structured data was added to a landing page and blog posts, and how a dynamic robots.txt file generation was replaced with a static one for better performance.

Implementing JSON-LD Structured Data

Structured data helps

Our application now supports content negotiation to better serve AI agents and LLMs. We've added the ability to return Markdown instead of HTML when requests include the Accept: text/markdown header.

This enhancement allows AI agents to directly consume the raw Markdown content of our posts, simplifying parsing and improving efficiency.

The Problem

Previously, our application served HTML