# Markdown for agents: serving Accept: text/markdown An agent asks for your page and gets HTML. Then it throws most of it away — the nav, the footer, the cookie banner, the inline scripts, the class attributes, the SVG icons — and keeps a few paragraphs of prose. You already have the version it wanted. Almost nobody serves it. ## What the markup costs Compare what you send with what survives: ```bash curl -s https://example.com/about | wc -c curl -s -H "Accept: text/markdown" https://example.com/about | wc -c ``` If both numbers are the same, you serve one representation to everything and the agent does the stripping itself. That work is not free. Everything inside ``, every wrapper `
`, every `data-` attribute and every analytics snippet is read into the context window before anything decides it was noise. On a page whose real content is a heading and eight paragraphs, most of the bytes are scaffolding. The Markdown twin keeps the heading levels, the links, the lists, the tables and the code blocks. It loses navigation, styling and script — none of which was ever going to be quoted. ## How Accept negotiation works The client says what it can use: ``` GET /about HTTP/1.1 Host: example.com Accept: text/markdown ``` The server picks a representation for the same URL: ``` HTTP/1.1 200 OK Content-Type: text/markdown; charset=utf-8 Vary: Accept ``` A browser sends `Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8` and gets HTML. A client that asks for `text/markdown` gets Markdown. The canonical URL does not change, so there is still one address to link and one page to share. ## Vary: Accept, or how a browser ends up downloading a text file `Vary` is not optional and it is the part people skip. Every cache between you and the reader — your CDN, a reverse proxy, the browser's own store — keys responses by URL. Serve two different bodies at one URL without telling it, and the first response cached wins for everybody. An agent asks for `/about` with `Accept: text/markdown`, your CDN stores the Markdown, and the next human to open `/about` gets a page of raw asterisks, or a download prompt. ``` Vary: Accept ``` That line tells the cache to include the `Accept` header in its key, so HTML and Markdown are stored separately. Send it on **both** responses, not just the Markdown one. There is a real cost: `Accept` headers vary a lot between clients, so varying on them fragments your cache. That is one honest argument for the alternative below. ## The .md twin The simpler option is a second URL. `/about` serves HTML; `/about.md` serves the same content as Markdown. No negotiation, no `Vary`, nothing to get wrong in a cache. It is a plain URL, so it can be linked, bookmarked, put in a sitemap and fetched by any client — including one that sends no useful `Accept` header at all. Then say it exists. In the HTML: ```html ``` And in the response headers, for anything that reads a `HEAD` and never parses the body: ``` Link: ; rel="alternate"; type="text/markdown" ``` Doing both costs one line each and covers clients that only look in one place. ## Checking yours ```bash # 1. Does the same URL answer differently? curl -s -H "Accept: text/markdown" https://example.com/about | head -20 # 2. Does the HTML response admit the twin exists? curl -sI https://example.com/about | grep -i "vary\|link" # 3. Does the twin URL work on its own? curl -sI https://example.com/about.md | grep -i "content-type\|vary" ``` The second command is the one that catches the common failure. A site can have a perfectly good `.md` twin that nothing will ever find, because no header and no tag points at it. ## What we do here Every page on this site has a Markdown twin, and `packages/web/src/lib/content-negotiation.js` is the whole of it — about ninety lines. A fixed map lists the static paths and their twins: `/` → `/index.md`, `/about` → `/about.md`, every tool page, and so on. Paths that carry an id or a slug — reports at `/r/`, guides, blog posts like this one — are matched by pattern instead and get `.md`. The request check is deliberately blunt: if the `Accept` header contains the string `text/markdown`, send Markdown. No q-value ranking. A browser never sends that string, so in practice it separates the two cleanly, but a client that lists `text/markdown` at a low priority gets Markdown anyway. Every Markdown response goes out with: ``` Content-Type: text/markdown; charset=utf-8 Content-Signal: search=yes, ai-input=yes, ai-train=no Vary: Accept X-Markdown-Tokens: 4213 Cache-Control: public, max-age=3600 ``` `Content-Signal` states what may be done with the content rather than only whether it can be fetched. `X-Markdown-Tokens` is a size marker — today it carries the character count of the body, not a real token count. HTML responses for a page with a twin get `Vary: Accept` and the `Link` header, and the `` gets the `rel="alternate"` tag from the same map. That last detail bit us. The map fed the template, but the pattern-matched routes were not in the map — so every analysis report served its Markdown twin and advertised it in the `Link` header, while its HTML said nothing. The twin existed and was invisible to anything reading the markup. ## It is a convention, not a standard `Accept: text/markdown` is ordinary HTTP content negotiation, which is a standard. Using it for Markdown is a convention that grew out of the llms.txt proposal, and no crawler is obliged to ask. The honest position is that the `.md` twin is the part that reliably works, because it is a URL like any other and anything can fetch it, while negotiation costs a header and pays off with the clients that do ask. Neither will hurt you, and you find out which is being used by reading your logs, not by trusting an announcement. [Test your Markdown negotiation](/tools/markdown-tester) · [Generate your llms.txt](/tools/llms-txt-generator)