Blog

An agent fetches your HTML and throws most of it away. How to hand it clean Markdown instead: negotiation, the Vary trap, .md twins, curl checks.

robots.txt says whether a bot may fetch a page. Content-Signal says what it may then do with it: index, answer, train. Three worked positions.

Complete robots.txt files to copy for the three cases that matter: let everything in, allow search but not training, or block AI entirely.

We measured twelve well-known pages with our own engine. On average 98% of what they send never reaches the model. Here is what survives, and why.

It is usually not the model. Five failure modes in the reading layer, measured on real pages — including the one where rendering the page times out.

Three things decide whether an AI assistant can quote you: robots.txt, JavaScript, and how much content survives extraction. Check all three yourself.

AI agents read your site through a different door than people do. AgentReady measures what they find and tells you what to change first.

Cloudflare's scanner checks which agent protocols your site declares. Ours checks whether your content survives being read. You want both answers.