Frequently Asked Questions

AI-readiness measures how well your website's content can be understood, extracted, and used by AI agents like ChatGPT, Claude, and Perplexity. As AI-powered tools become a major source of web traffic, sites that are AI-ready get cited more accurately, appear more often in AI-generated responses, and cost less tokens to process.

Unlike web browsers that render HTML visually, AI agents need to extract text content from your pages. They prefer clean, well-structured content over complex HTML with heavy styling. A well-structured page converted to Markdown uses 70-80% fewer tokens than raw HTML, making it cheaper and more efficient for AI providers.

The major AI crawlers include GPTBot (OpenAI/ChatGPT), ClaudeBot (Anthropic/Claude), PerplexityBot (Perplexity), Google-Extended (Google Gemini), Bytespider (ByteDance), CCBot (Common Crawl), and many more. New AI agents appear regularly as the ecosystem grows.

llms.txt is an emerging standard (defined at llmstxt.org) that helps AI agents understand your website's structure. Similar to how robots.txt guides search engine crawlers, llms.txt provides a Markdown-formatted overview of your site with links to key pages, making it easy for AI agents to navigate your content.

llms.txt is a concise index with a description and links to your site's main pages. llms-full.txt is an extended version that includes the actual content of those pages inline, giving AI agents everything in a single file without needing to follow links. Use llms.txt as a minimum, and llms-full.txt for comprehensive coverage.

Create a text file at your domain root (e.g., example.com/llms.txt) following the llmstxt.org spec. Start with a # heading (your site name), add a blockquote description, then list links organized in sections like ## Documentation and ## Main. AgentReady can generate a recommended llms.txt based on your page analysis.

Markdown is the preferred format for AI agents because it preserves content structure (headings, lists, links, emphasis) while eliminating visual markup noise (CSS, JavaScript, layout divs). A Markdown version of your content uses significantly fewer tokens, making it faster and cheaper for AI systems to process.

Content negotiation allows your server to serve different formats of the same page based on the client's Accept header. When an AI agent sends Accept: text/markdown, your server can respond with a Markdown version instead of HTML. This is the most efficient way to serve AI-friendly content without creating separate URLs.

There are two main approaches: (1) Add server logic to detect Accept: text/markdown headers and return Markdown content; (2) Create .md files alongside your pages (e.g., /about.md for /about) and link to them from your llms.txt. AgentReady uses both approaches for its own pages.

JSON-LD (JavaScript Object Notation for Linked Data) is a way to embed structured data in your pages using Schema.org vocabulary. AI agents use this data to extract factual, machine-readable information like product details, article metadata, organization info, and more — without needing to parse your HTML.

Use the most specific type that matches your content: Article or BlogPosting for articles, Product for product pages, Organization for company pages, FAQPage for FAQ pages, LocalBusiness for local businesses, and WebApplication for web tools. Always include name, description, and relevant properties for your chosen type.

Open Graph tags (og:title, og:description, og:image) provide standardized metadata that both social platforms and AI agents use to understand your page's title, description, and primary image. They're easy to implement and serve as a reliable fallback when other structured data is missing.

robots.txt controls which bots can access your site and which pages they can crawl. AI crawlers like GPTBot and ClaudeBot respect robots.txt directives. If your robots.txt blocks these bots, they won't be able to index your content, which means your site won't appear in AI-generated responses.

To maximize visibility in AI-generated responses, allow at least: GPTBot (OpenAI), ClaudeBot and Claude-Web (Anthropic), PerplexityBot (Perplexity), and Google-Extended (Google Gemini). You can add specific Allow rules for these user agents while maintaining your existing rules for other bots.

Content-Signal is an HTTP header that tells AI agents how they may use your content. For example: Content-Signal: ai-train=yes, search=yes, ai-input=yes signals that your content can be used for AI training, search indexing, and as input for AI responses. This is a newer standard that gives publishers explicit control over AI usage.

AgentReady fetches your page, extracts the content, and runs 23 individual checks across 5 weighted dimensions. Each check scores 0-100, and the dimensions are combined into an overall score from 0 to 100. You get a letter grade (A-F), detailed breakdown, and prioritized recommendations to improve your score.

The 5 dimensions are: Semantic HTML (15%) — proper use of article, main, headings, and semantic elements; Content Efficiency (10%) — token reduction ratio and content-to-noise ratio; AI Discoverability (25%) — llms.txt, robots.txt, sitemap, and markdown negotiation; Structured Data (20%) — Schema.org, Open Graph, and meta tags; Accessibility (30%) — content without JavaScript, page size, and content position.

Yes! Single-page analysis is completely free with no signup required. You get the full score, recommendations, Markdown conversion, and llms.txt preview. We're currently in beta with a limit of 5 analyses per hour. Full domain crawl and monitoring features are coming soon.

We check 18 crawlers, and every one of them links to its vendor's own documentation. In July 2026 we rebuilt the list. Three entries were retired: Claude-Web and FacebookBot no longer appear in Anthropic's or Meta's docs, and cohere-ai was never documented at all — we had been telling people to name bots that do not exist. We also added the ones that were missing, including OAI-SearchBot, the crawler that actually puts your site in ChatGPT's search results. Third-party crawler lists circulate invented user-agents (the xAI/Grok ones are a known case). We only list a bot if its vendor documents it.

No — and this distinction is the single most useful thing on this page. Vendors split their crawlers by purpose. Training bots (GPTBot, ClaudeBot, Google-Extended) read your pages to train models: blocking them costs you no visibility whatsoever. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index the assistant cites from: block one and you disappear from that assistant. User bots (ChatGPT-User, Claude-User) fetch a page because a person asked for it right now. You can refuse training and still be cited everywhere — but only if you allow the search bots. We now label every crawler by kind so you can make that choice deliberately instead of blocking everything with one wildcard.

The A2A agent card belongs at /.well-known/agent-card.json. It moved there in A2A v0.3.0 (August 2025) and v1.0 keeps it; /.well-known/agent.json is a deprecated alias that SDKs still serve for old clients. We were checking — and generating — the old path, which meant publishing a descriptor where current agents do not look. That is fixed. For MCP the honest answer is that there is no standard path: the MCP specification defines no well-known discovery file at all, and server cards are still an open draft (SEP-1649) proposing /.well-known/mcp/server-card.json. We probe both that and the older mcp.json convention, and we label them a draft rather than pretending otherwise. What the MCP spec does require is /.well-known/oauth-protected-resource (RFC 9728) — we now check for it.

Because the rubric rewarded what is easy to check rather than what we can show matters. llms.txt — a proposal no major vendor has committed to implementing — outweighed Schema.org, which every assistant actually parses. Rubric v2 makes each of the 23 checks declare its evidence: proven (the vendor documents that it reads this), plausible (long-established web practice, but no vendor statement about AI specifically), or speculative (a reasonable bet, nothing more). The weights follow that: proven now carries 32 of the 100 points, plausible 51, speculative 17. We still check llms.txt — it costs nothing to publish and it may yet matter — but it no longer outranks the signals that demonstrably do. Every result records the rubric version that produced it, so an older score can be read against the rules it was measured with.

We measure it now, and it exposed a hole in our own scoring. Every other check reads a declaration: your robots.txt says the crawlers may pass, so we awarded the points. But robots.txt is served by your origin, and the request never gets there — your CDN or WAF answers first, and it does not have to agree. Cloudflare began blocking AI crawlers by default at the edge on 15 September 2026. A site could publish a flawless robots.txt, be shut to every assistant, and score full marks from us. So we stopped reading and started measuring: we fetch your page as an ordinary client, fetch it again announcing ourselves as OAI-SearchBot, and compare. What we can prove has limits, and we would rather say so than overclaim. Our probe comes from our servers, and edges verify crawlers by IP address, not by name — so a refusal might be aimed at impostors while the verified crawler passes. We report exactly what we saw and point you at your bot rules. We do not tell you that you are blocked from ChatGPT, because from outside we cannot know that.