Markdown for agents: serving Accept: text/markdown
An agent asks for your page and gets HTML. Then it throws most of it away — the nav, the footer, the cookie banner, the inline scripts, the class attributes, the SVG icons — and keeps a few paragraphs of prose.
You already have the version it wanted. Almost nobody serves it.
What the markup costs
Compare what you send with what survives:
curl -s https://example.com/about | wc -c
curl -s -H "Accept: text/markdown" https://example.com/about | wc -c
If both numbers are the same, you serve one representation to everything and the agent does the stripping itself. That work is not free. Everything inside <head>, every wrapper <div>, every data- attribute and every analytics snippet is read into the context window before anything decides it was noise. On a page whose real content is a heading and eight paragraphs, most of the bytes are scaffolding.
The Markdown twin keeps the heading levels, the links, the lists, the tables and the code blocks. It loses navigation, styling and script — none of which was ever going to be quoted.
How Accept negotiation works
The client says what it can use:
GET /about HTTP/1.1
Host: example.com
Accept: text/markdown
The server picks a representation for the same URL:
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
A browser sends Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 and gets HTML. A client that asks for text/markdown gets Markdown. The canonical URL does not change, so there is still one address to link and one page to share.
Vary: Accept, or how a browser ends up downloading a text file
Vary is not optional and it is the part people skip.
Every cache between you and the reader — your CDN, a reverse proxy, the browser's own store — keys responses by URL. Serve two different bodies at one URL without telling it, and the first response cached wins for everybody. An agent asks for /about with Accept: text/markdown, your CDN stores the Markdown, and the next human to open /about gets a page of raw asterisks, or a download prompt.
Vary: Accept
That line tells the cache to include the Accept header in its key, so HTML and Markdown are stored separately. Send it on both responses, not just the Markdown one.
There is a real cost: Accept headers vary a lot between clients, so varying on them fragments your cache. That is one honest argument for the alternative below.
The .md twin
The simpler option is a second URL. /about serves HTML; /about.md serves the same content as Markdown. No negotiation, no Vary, nothing to get wrong in a cache. It is a plain URL, so it can be linked, bookmarked, put in a sitemap and fetched by any client — including one that sends no useful Accept header at all.
Then say it exists. In the HTML:
<link rel="alternate" type="text/markdown" href="https://example.com/about.md">
And in the response headers, for anything that reads a HEAD and never parses the body:
Link: <https://example.com/about.md>; rel="alternate"; type="text/markdown"
Doing both costs one line each and covers clients that only look in one place.
Checking yours
# 1. Does the same URL answer differently?
curl -s -H "Accept: text/markdown" https://example.com/about | head -20
# 2. Does the HTML response admit the twin exists?
curl -sI https://example.com/about | grep -i "vary\|link"
# 3. Does the twin URL work on its own?
curl -sI https://example.com/about.md | grep -i "content-type\|vary"
The second command is the one that catches the common failure. A site can have a perfectly good .md twin that nothing will ever find, because no header and no tag points at it.
What we do here
Every page on this site has a Markdown twin, and packages/web/src/lib/content-negotiation.js is the whole of it — about ninety lines.
A fixed map lists the static paths and their twins: / → /index.md, /about → /about.md, every tool page, and so on. Paths that carry an id or a slug — reports at /r/<uuid>, guides, blog posts like this one — are matched by pattern instead and get <path>.md.
The request check is deliberately blunt: if the Accept header contains the string text/markdown, send Markdown. No q-value ranking. A browser never sends that string, so in practice it separates the two cleanly, but a client that lists text/markdown at a low priority gets Markdown anyway.
Every Markdown response goes out with:
Content-Type: text/markdown; charset=utf-8
Content-Signal: search=yes, ai-input=yes, ai-train=no
Vary: Accept
X-Markdown-Tokens: 4213
Cache-Control: public, max-age=3600
Content-Signal states what may be done with the content rather than only whether it can be fetched. X-Markdown-Tokens is a size marker — today it carries the character count of the body, not a real token count.
HTML responses for a page with a twin get Vary: Accept and the Link header, and the <head> gets the rel="alternate" tag from the same map.
That last detail bit us. The map fed the template, but the pattern-matched routes were not in the map — so every analysis report served its Markdown twin and advertised it in the Link header, while its HTML said nothing. The twin existed and was invisible to anything reading the markup.
It is a convention, not a standard
Accept: text/markdown is ordinary HTTP content negotiation, which is a standard. Using it for Markdown is a convention that grew out of the llms.txt proposal, and no crawler is obliged to ask. The honest position is that the .md twin is the part that reliably works, because it is a URL like any other and anything can fetch it, while negotiation costs a header and pays off with the clients that do ask. Neither will hurt you, and you find out which is being used by reading your logs, not by trusting an announcement.