Blog
Notes on how AI agents read the web, and what to do about it.
-
How we score: 23 checks, and why some weigh more than others
The full rubric: five weighted dimensions, 23 checks and the evidence tier behind each one. Including what the score deliberately leaves out.
Read -
Making PM2 survive a server reboot (both halves)
A server that only comes back when someone deploys is missing pm2 save or pm2 startup. How to install both, verify each, and test it early.
Read -
Cache busting: why your ?v= probably does nothing
A stale stylesheet broke our navigation and looked exactly like a failed deploy. Why only a content hash works as a cache-busting token.
Read -
The canonical mistake that hides your translations
We shipped a site in ten languages where nine told search engines not to index them. The bug, the rule that prevents it, and the one-line check.
Read -
Why your CDN caches nothing: the header that blocks it
Every asset came back cf-cache-status: BYPASS while all four cache settings were correct. The cause was a Set-Cookie on every response.
Read -
Markdown for agents: serving Accept: text/markdown
An agent fetches your HTML and throws most of it away. How to hand it clean Markdown instead: negotiation, the Vary trap, .md twins, curl checks.
Read -
Content-Signal: telling AI what it may do with your content
robots.txt says whether a bot may fetch a page. Content-Signal says what it may then do with it: index, answer, train. Three worked positions.
Read -
How to allow or block AI bots in robots.txt
Complete robots.txt files to copy for the three cases that matter: let everything in, allow search but not training, or block AI entirely.
Read -
What an AI agent actually sees when it visits your site
We measured twelve well-known pages with our own engine. On average 98% of what they send never reaches the model. Here is what survives, and why.
Read -
Why your AI agent keeps failing on real websites
It is usually not the model. Five failure modes in the reading layer, measured on real pages — including the one where rendering the page times out.
Read -
Can ChatGPT, Claude and Perplexity read your site?
Three things decide whether an AI assistant can quote you: robots.txt, JavaScript, and how much content survives extraction. Check all three yourself.
Read -
What AgentReady is, and what it actually tells you
AI agents read your site through a different door than people do. AgentReady measures what they find and tells you what to change first.
Read -
isitagentready.com and AgentReady.md: what each one measures
Cloudflare's scanner checks which agent protocols your site declares. Ours checks whether your content survives being read. You want both answers.
Read